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PREFACE 


This  report  was  prepared  as  part  of  Rand's  OSD  Manpower,  Personnel, 
and  Training  Program,  sponsored  by  the  Human  Resources  Research  Office 
of  the  Defense  Advanced  Research  Projects  Agency  (ARPA) .  With  manpower 
considerations  assuming  an  ever-increasing  importance  for  the  Departj 
ment  of  Defense,  the  purpose  of  this  research  program  is  to  bring  new 
methodologies  to  bear  on  present  and  future  military  manpower  problems. 

The  report  describes  a  prototype  computer  program  based  on  "admis¬ 
sible  probability  measurement"— a  testing  procedure  which  allows  a  user 
to  express  his  uncertainty,  in  terms  of  probabilities,  about  the  cor¬ 
rectness  of  his  answers  to  multiple-choice  questions.  A  logarithmic 
scoring  system  encourages  the  user  to  "reproduce"  or  "reveal"  his  per¬ 
sonal,  subjective  probability  of  correctness  for  each  of  the  choices. 

The  on-line,  graphical  nature  of  the  computer  program  provides  an  ex¬ 
cellent  environment  for  allowing  the  user  to  interact  with  such  a 
scoring  system. 

Admissible  probability  measurement  must  be  considered  carefully 
in  terms  of  the  environment  in  which  it  is  administered  and  its  possible 
effects  upon  users.  These  issues,  among  others,  are  discussed  in  a 
companion  report,  R-1371-ARPA,  A  Rationale  and  Some  Implications  of 
Computer- Administered  Admissible  Probability  Measurement  by  Emir 
Shuford,  Jr.  and  Thomas  A.  Brown  (forthcoming). 

This  report  should  be  useful  to  administrators  concerned  with 
advanced  testing  methods,  to  instructors  of  courses  involving  the 
estimation  of  probabilities,  and  to  computer  programmers  who  wish  to 
implement  a  version  of  computer-administered  admissible  probability 
measurement .  J 


SUMMARY 


This  report  describes  an  experimental  computer  implementation  of 
admissible  probability  measurement  techniques.  These  measurement  tech¬ 
niques  rely  on  a  scoring  system  that  encourages  the  user  to  "reproduce" 
or  "reveal"  his  own  estimate  of  the  probability  of  correctness  of  an: 
answer  to  a  multiple-choice  question. 

A  "track  record"  (in  the  form  of  an  external  validity  graph)  of 
the  user’s  probability  estimating  performance  provides  the  means  for 
estimating  how  much  information  he  has  and  how  realistically  that  level 
of  knowledge  is  assessed. 

The  computer  system  allows  the  user  to  explore  the  relationships 
between  his  assignment  of  probability  of  correctness  to  answers  and  the 
possible  score  outcome.  In  addition,  the  computer  analyzes  the  ex¬ 
ternal  validity  graph  generated  during  test  administration  to  provide 
feedback  about  his  present  state  of  knowledge  and  how  realistically  that 
knowledge  is  applied.  The  ease  of  use  of  the  computer  system  and  the 
speed  of  analysis  permit  cycling  through  tests  quickly,  thus  removing 
a  major  barrier  from  the  task  of  learning  to  make  better  assessments  of 
the  user’s  knowledge. 
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1.  INTRODUCTION 


I  PURPOSE  OF  THE  REPORT 

It  is  the  purpose  of  this  report  to  document  a  collection  of  com¬ 
puter  procedures  for  administering  admissible  probability  measurement 
(APM)  techniques.  The  report  is  divided  into  the  following  parts: 

•  An  introduction  to  the  motivations  and  mathematical 
foundations  behind  APM. 

•  A  description  of  the  environment  Cboth  hardware  and 
software)  in  which  the  computer  procedures  were  de¬ 
veloped,  and  a  description  of  the  operation  of  the 
computer  procedures. 

•  A  short  discussion  of  planned  modifications  and  ex¬ 
tensions  of  the  present  system. 

•  A  brief  introduction  to  the  PLATO  IV  system  at  the 
University  of  Illinois.  (A  version  of  APM  is  planned 
for  the  PLATO  IV  system.) 

•  Two  appendices  containing  detailed  descriptions  of 
support  software  as  well  as  listings  of  the  computer 
procedures. 


This  report  is  sufficiently  complete  to  allow  an  interested  in¬ 
vestigator  to  develop  his  own  operating  version  of  computer--administered 
APM. 


MOTIVATIONS  AND  FOUNDATIONS 

!  The  prime  motivation  for  APM  is  simple  and  straightforward:  We 

need  to  measure  more  accurately  the  "state  of  knowledge"  of  students 

in  a  large  variety  of  subject  areas.  An  improved  measuring  technique 

would  be  of  considerable  value  not  only  in  testing  a  student  but  also 

(X  2)  (2) 

in  evaluating  and  planning  curricula.  9  As  indicated  by  Gardner, 

| planning  is  especially  critical  for.  those  courses  of  instruction  thatj 
lare  individually  tailored  to  the  student.  ' 


A,  second  but  still  important  motivation  is  that  the  estimation  j 
of;  probabilities  is  a  critical  skill  in  some  endeavors,  such  as  weather 
prediction  and  intelligence  analysis.  APM  is  especially  applicable  ih 
jthe  training  of  students  to  use  effectively  all  of  their  facts  and 
'reasons  in  the  estimation  of  probabilities.  j 

A  corollary  of  the  need  to  train  specialists  in  probability  esti¬ 
mation  is  the  idea  that,  in  fact,  we  are  all  engaged  in  probability 
estimation.  That  is,  there  is  a  need  for  us  all  to  convey  accurately 
and  effectively  our  opinions  on  the  possibilities  of  various  outcomes. 

If  that  is  the  case,  we  must  learn  to  speak  the  language  of  probabilities 
precisely  and  with  a  common  understanding. 

So  what  is  admissible  probability  measurement  and  how  is  it  applied? 
AJPM  will  be  discussed  in  terms  of  one  specific  application  even  though 
it  has  greater  generality  than  may  be  apparent  from  this  discussion. 

Consider  the  multiple-choice  test  item  below. 

QUESTION:  The  11th  president  of  the  United  States  was: 

1.  George  Washington 

2.  James  Polk 

3.  Franklin  Pierce 

We  are  generally  required  to  respond  to  such  a  question  with  what 
amounts  to  a  probability  distribution,  as  illustrated  below. 


QUESTION:  The  11th  president  of  the  United  States  was: 


1.  George  Washington 

2.  James  Polk 

3.  Franklin  Pierce 


Probability 

Distribution 


However,  a  more  general  tendency  would  be  to  respond  with  the  proba¬ 
bility  distribution  shown  here. 
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QUESTION:  The  ll£7z  president  of  the  United  States  was: 


1 

[ 

1 
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1.  George  Washington 

! 
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1/2 

2 .  James  Polk 

i 

1/2 

3.  Franklin  Pierce 

Revised 

Probability 

Distribution 

In  fact,  if  confronted  with  the  following  question,  a  response 
might  very  well  be,  T?X  can’t  distinguish  among  the  alternatives."  That 
is,  "as  far  as  I  can  tell  from  my  facts  and  reasons,  each  alternative 
is  equally  likely." 

QUESTION:  The  11th  president  of  the  United  States  was: 

1.  John  Tyler 

2.  James  Polk 

3.  Franklin  Pierce 

It  is  evident  that  requiring  responses  of  0  or  1  can  mask  a  large 
variety  of  states  of  knowledge  and  can  introduce  guessing  on  the  part 
of  the  student. 

However,  if  greater  freedom  of  response  is  allowed,  there  remains 
the  problem  of  how  to  score  such  a  response.  That  is  precisely  the 
function  of  the  "admissible,"  or  "proper,"  or  "reproducing"  scoring 

(3) 

system.  Such  a  scoring  system  encourages  the  strategy  of  respond¬ 

ing  with  exactly  what  is  perceived  to  be  the  probability  distribution, 
based  on  the  user’s  own  facts  and  reasons. 

ADMISSIBLE  SCORING  SYSTEMS 

The  admissible  scoring  system  can  be  explained  in  terms  of  a 
gambling  example  (a  more  complete  discussion  can  be  found  in  Ref.  3). 
For  the  sake  of  simplicity,  suppose  that  a  student  were  faced  with  a 

; true-false  question.  In  that  situation  he  may  be  viewed  as  a  gambler 

! 

j faced  with  determining  the  probability  that  the  correct  response  is  | 
["true."  Suppose  also  that  there  were  a  gambling  house  in  which  there! 


1/3" 

1/3 

1/3 


j=A- 


I  were  <|>Cu)du  wagers  available  at  odds  Cl  -  u)/u  (the  "correct:;  odds11  for 
an  event  of  probability  u) .  If  the  student  wished  to  maximize  his  j 
expected  return  and  believed  that  the  probability  of  the  event  taking! 
j place  were  p,  he  would  take  all  wagers  at  odds  better  than  (1  -  p)/p. 

I  Similarly,  he  would  take  all  wagers  on  the  event  not  taking  place  at 
odds  better  than  those  appropriate  for  probability  1  -  p. 

The  mathematics  of  admissible  scoring  systems  are  illustrated 
below.  Equation  (1)  corresponds  to  our  true-or-false  discussion. 
Equation  (2)  extends  those  notions  to  an  n-alternative  multiple- choice 
question.  Equation  (3)  adjusts  the  equations  to  give  a  score  of  zero 
for  the  "unable  to  distinguish"  response  of  =  —  for  i  =  l,n.  What 
these  integrals  represent  is  the  net  settlement  made  between  the 
gambling  house  and  the  student,  after  the  correct  alternative  has  been 
revealed.  It  is  the  sum  of  settlements  on  an  infinite  number  of  in¬ 
finitesimal  wagers,  each  of  which  appeared  to  the  student  to  be  favor¬ 
able  to  him. 


True  or  false 


payoff  (if  "true"  occurs) 


( j)  (u)  du 


Cl) 


Multiple  choice  (n  alternatives) 


payoff 

if  1th  event  occurs)  - 


(ftCu)du 

u 


n 


E 


<f)(u)du 
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Adjusted  multiple  choice 


payoff 

(if  1th  event  occurs) 


<Ku)du  _  J 


j=l  1/n 


<f)(u)du 


(3) 


r  Equations  (4)  and ~(5T below  illustrate  the  logarithmic  admissibly 
I  scoring  system  which  has  the  property  of  depending  only  on  the  proba-j 
ibility  ascribed  to  each  alternative  independent  of  the  probabilities  j 
ascribed  to  the  remaining  alternatives. 


The  logarithmic  scoring  system  <j>(u)  =  1: 


payoff 

(if  1th  event  occurs) 


=  log(p±)  -  log(±)  -  CPj 

j=l 


n 


log(np^)  (assuming  ^  ^  p_.  =  1) 


(4) 


An  adjusted  system  for  three  alternatives  and  a  score  range  of 
about  100  points: 


=  50  log(3p±)  0.01  £  p±  £  0.98  (5) 

Equation  (5)  illustrates  the  particular  version  of  the  logarithmic 
system  we  use  in  APM.  To  avoid  infinitely  large  penalties,  no  proba¬ 
bility  less  than  0.01  may  be  assigned! 

The  gambling  illustration  shows  that  it  is  in  the  student's  best 
interest  to  gauge  his  probabilities  accurately  and  thus  to  make  no  un¬ 
necessary  bets.  That  is,  the  scoring  system  does  encourage  him  to 
reproduce  his  own  probabilities  accurately. 

However,  how  does  the  student  learn  to  "play  the  game"  and  improve 
the  way  he  brings  his  facts  and  reasons  to  bear  on  the  determination  of 
the  probability  he  ascribes  to  each  alternative?  As  Gardner  points 
out,  "It  seems  likely  that  the  more  exposure  the  students  have  to  [proba 
bility]  tests,  the  more  realistically  they  will  evaluate  t esf  items,  and 


j\ 


their  own  knowledge,  thus  providing  a  more  valid  indication  as  to  their 
actual  level  of  achievement."^  We  believe  the  answer  to  the  question 
lies  in  the  computer  system  that  will  be  described  next. 

THE  ROLE  OF  THE  COMPUTER 

As  can  be  seen  in  the  figures  that  follow,  the  computer-based  sys¬ 
tem  we  have  developed  requires  a  computer  terminal  with  graphics  capa¬ 
bilities  and  some  means  of  allowing  the  student  to  interact  with  it  in 
elapsed  times  of  at  most  a  few  seconds.  The  system  was  developed  on  a 
highly  interactive  console  connected  to  a  powerful  1BH  360/65  computer. 

Figure  1  illustrates  the  display  the  student  sees  while  answering 
a  question.  He  assigns  a  probability  to  each  alternative  by  choosing 
a  point  in  the  interior  or  on  the  boundary  of  the  equilateral  triangle. 

In  such  a  triangle,  the  sum  of  the  distances  of  the  point  from  each 
side  of  the  triangle  is  a  constant  and  can  be  scaled  to  1.  Thus  each 
point  determines  a  probability  distribution  for  the  three  alternatives. 
The  probabilities  are  therefore  naturally  constrained  to  sum  to  1.  The 
logarithmic  scores  corresponding  to  each  alternative  appear  as  both 
numeric  scores  and  columns  of  dots.  The  labels  A,  B,  and  C  create  the 
correspondences  between  the  alternatives,  the  triangle  vertices,  and 
the  scores.  The  student  may  choose  different  points  within  the  triangle 
until  he  is  satisfied  with  his  score  distribution.  He  may  then  select 
the  QUESTION  ANSWER  option  and  the  terminal  displays  his  cumulative 
score  as  illustrated  by  Fig.  2. 

The  cumulative  score  is  presented  in  the  form  of  the  maximum  (or 
minimum)  score  attainable,  question  by  question  (Fig.  2).  A  probability 
of  0.98  assigned  to  a  correct  answer  would  result  in  a  gain  of  24  points, 
and  0,98  assigned  to  an  incorrect  alternative  would  result  in  a  loss  of 
76  points.  Thus,  the  student  has  question-by-question  feedback  about 
the  relationship  of  his  probability  assignments  to  his  score  and  may 
adjust  his  actions  accordingly.  That  is,  he  may  develop  a  "feel"  for 
the  scoring  system.  Under  usual  test  administration  procedures,  the 
delay  from  testing  to  reporting  the  scores  is  so  long  that  it  prevents 
|the  development  of  a  "feel"  for  the  scoring  system.  Moreover,  the  j 
scores  are  usually  not  reported  on  an  item  by  item  basis. _ This  portion 
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Fig.  1  —  Display  for  a  question 


pig.  2  —  Cumulative  score  for  a  question 


of  the  computer-based  system  gives  the  student  a  maximum  amount  of  ; 
experience  with  the  scoring  system  and  its  immediate  consequences.  | 

Even  though  the  student  may  learn  to  express  his  own  probabilities 
accurately,  the  question  remains  as  to  how  those  probabilities  relate 
! to  reality.  That  question  can  be  answered  in  part  by  his  external 

[validity  graph. ^ 

i 

i 

‘the  external  validity  graph 

The  external  validity  graph  can  be  used  to  determine  how  accurately 
■the  student  assesses  the  predictive  validity  of  his  facts  and  reasons. 

It  can  be  thought  of  as  his  "track  record"  as  an  assessor.  We  may 
collect  all  those  events  that  he  claims  have  a  probability  of,  say, 

0.80  of  occurring  and  determine  what  percentage  of  them  did  in  fact 
occur.  If  he  were  a  perfect  assessor,  we  would  expect  roughly  80  per¬ 
cent  of  them  to  occur.  Figure  3  illustrates  such  a  graph  when  the 
number  of  observations  is  unlimited.  The  diagonal  AB  represents  a 
perfect  assessor.  If  we  collect,  say  for  20  questions,  all  the  proba- 
-bilities  used  by  a  student  and  compute  the  relative  frequency  with 
which  each  particular  probability  is  associated  with  a  correct  alter¬ 
native,  we  may  construct  an  external  validity  graph  for  that  student 
taking  that  test.  We  then  assume  that  the  relationship  between  student 
; response  and  relative  frequency  can  be  approximated  satisfactorily  by 
a  straight  line  simple  linear  regression. 

We  call  the  resulting  line  the  student's  realism  line.  The  lines 
' A f B 1  and  A"B"  in  Fig.  3  illustrate  possible  results.  The  line  AfB* 
has  the  following  interpretation:  About  60  percent  of  the  events  that 
.were  given  a  probability  of  1.0  of  occurring  did  in  fact  occur.  We 
| interpret  this  to  mean  that  the  student  tends  to  overvalue  his  facts 
;and  reasons.  Similarly,  interpretation  of  the  line  A"B"  shows  that 
■the  events  that  were  given  a  probability  of  0.60  (or  greater)  of  occur¬ 
ring  did  in  fact  occur  100  percent  of  the  time.  In  other  words,  the 
I  student  tends  to  undervalue  his  facts  and  reasons.  We  may  further 
interpret  the  realism  line  in  the  following  way:  "When  he  responded  x 
he  should  have  responded  with  Ax  +  B,"  where  Ax  +  B  is  the  equation  of 
his  realism  line.  We  may  then  transform  the  student's  probabilities,; 


Fig.  3  —  External  validity  graph  and  realism  lines 


using  his  realism  line,  and  recalculate  his  score  on  the  basis  of  the 
new  probability  assignments.  This  revised  score  can  then  be  used  to 
decompose  his  original  score  into  the  portion  attributable  to  bias  in 
his  assignment  of  probabilities  and  the  portion  attributable  to  lack 
of  information  about  the  subject  matter  itself.  The  lower  section  of 
Fig.  4  illustrates  such  a  score  decomposition. 

The  scores  are  actually  reported  as  the  difference  between  the 
studentfs  scores  and  the  maximum  attainable  for  a  test  with  a  specified 
length.  The  overall  gain  is  that  maximum  less  the  original  score.  The 
gain  a  t  taihable  frolnhmore  information  is  t  ha  t  maximum  less  the  revised 
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Eig.  4  —  Score  interpretation  display 


score.  The  gain  from  more  "realistic"  use  of  probabilities  is  simply 
the  revised  score  less  the  original  score. 

i  : 

j  INFORMATION  THEORETIC  MEASURES 

The  upper  portion  of  Fig.  4  brings  us  to  another  interesting  as¬ 
pect  of  the  logarithmic  scoring  system.  We  call  the  square  in  the 
upper  part  of  Fig.  4  the  student’s  information  square.  The  term  "in-: 
|  formation"  arises  from  a  very  interesting  relation  between  the  log- 
jarithmic  scoring  system  and  the  concept  of  "entropy"  in  information 

'  ■  J  ; 

'theory.  That  relation  is  developed  below. 

Equation  (6)  below  is  the  expected  score  for  a  given  question  in 
our  three  alternative  systems. 

3 

Expected  score  =  p_^(50  l°g  ^  P^) 

i=l 


=  50 


log  3 


3 

o  ^2  Pi  log  P;L) 

i=l 


(6) 


Equation  (7)  is  called,  in  information  theory,  the  "entropy"  of  the 
'partition  p^.  It  represents  the  expected  amount  of  information  which 
will  be  conveyed  by  the  event  itself;  or,  in  other  words,  the  expected 
surprise  in  finding  which  alternative  is  correct. 


Entropy  = 


3 


i=l 


(7) 


Equation  C8)  is  the  expected  score  normalized  to  the  range  0  to  1. 


50 


log  3  -  (-  ^2  Pi  loS  P-j) 


i=l 


50  log  3 


(8) 


Relative  information  = 


=3 J3a_ 


jEquatlon  (8)  can  be  interpreted  as  follows:  l;f  there  is  no  surprise  j 
'content  in  the  event  (e.g.,  p-^  =  1,  P2  "  0>  P3  =  05  then  there  is  pery 
Ifect  information  (jEq.  C8)  =1);  if  there  is  maximum  surprise  content  I 

jin  the  event  Ci-e.,  Pi  =  ?2  =  p3  =  then  there  is  no  information 

;(Eq.  C8)  =  0).  The  upper  part  of  Fig.  4  plots  on  the  PERCEIVED  axis  . 
the  points  corresponding  to  the  relative  information  computed  from  the 
student's  original  probability  distributions.  The  relative  informa¬ 
tion  derived  from  the  probabilities  as  transformed  by  the  realism  line 
(the  regression  line  in  the  external  validity  graph)  is  plotted  on  the 
REVISED  axis  and  the  two  points  are  connected  by  a  line.  This  infor¬ 
mation  square  has  an  interesting  interpretation  in  terms  of  a  proverb 
as  depicted  in  Fig.  5. 


Wise 


Child 


He  who  knows,  and  knows  that-  he  knows. 

He  is  wise,  follow  him. 

He  who  knows,  and  knows  not  that  he  knows. 

He  is  asleep,  awaken  him. 

He  who  knows  not,  and  knows  not  that  he  knows  not. 
He  is  a  fool,  shun  him. 

He  who  knows  not,  and  knows  that  he  knows  not, 

He  is  a  child,  teach  him. 


Fig.  5  —  A  proverb 
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II .  COMPUTED  SYSTEM  DESCRIPTION 


1  HARDWARE 

i  The  current  version  of  the  computer-administered  probability  mea- 

| surement  program  runs  on  The  Rand  Corporation’s  IBM  360  Model  65  com¬ 
puter,  which  is  coupled  to  user  consoles  (see  Fig.  6)  via  the  Rand  Video 

(4) 

Graphic  System.  Each  videographic  console  consists  of  a  studio- 

; monitor-quality,  black-and-white  television  set,  a  keyboard,  and  a  hand¬ 
held  light  pen.  The  videographic  light  pen  is  unique  in  the  sense  that 
it  counts  television  scan  lines  and  light  positions  within  those  scan 
lines  to  report  x,y  positions  within  the  display.  This  x,y  position 
is  transmitted  from  the  console  to  the  IBM  360/65  via  the  videographic 
system  and  is  there  compared  to  known  positions  or  figures  on  the  dis¬ 
play.  Pictures  are  constructed  within  the  IBM  360/65  and  transmitted 
to  the  consoles  again  via  the  videographic  system.  Figures  7a  and  7b 
provide  a  system  diagram  for  Rand’s  Video  Graphic  System. 

SUPPORT  SOFTWARE 

The  IBM  360/65  operates  under  IBM’s  operating  system  with  the  MVT 
(multi-tasking  with  a  variable  number  of  tasks)  option.  The  operating 
system  software  has  been  extended  by  Rand  to  include  the  video  operat¬ 
ing  system ^  which  accommodates  the  message  traffic  between  the  360/65 
and  the  videographic  terminals. 

The  probability  measurement  program  is  written  in  FORTRAN  IV.  The 
communication  between  the  program  and  the  videographic  system  is  handled 
by  a  FORTRAN-callable  graphics  package  called  VPACK, v  }  which  was  created 
especially  for  the  Rand  Video  Graphic  System.  (Appendix  A  extracts  the 
features  of  VPACK  used  by  the  measurement  program.) 

The  present  version  of  the  measurement  program  does  not  allow  for 
multiple  consoles.  It  is  submitted  as  a  standard  computer  job  which 
occupies  an  operating  system  partition  for  as  long  as  testing  continues. 

A  multiple-console  version  would  require  additional  hardware  {more  light 
jpens)  and  extensive  software  development  (a  timesharing  subsystem  in 
i  IBM’  s  OS/ 3 60) .  _ _  _  _ _ j 


v-^ 


Fig.  6 


Video  Graphic  console 


SPECIFIC  SOFTWARE  ,  i 

fi-  l 

The  basic  organization  of  the  measurement  program  is  illustrated: 
in  the  state  diagram  in  Fig.  8.  The  names  within  parentheses  (for  ex¬ 
ample,  MSELECTM)  are  the  names  of  the  subroutines  called  by  the  main 
program.  (A  complete  listing  of  the  main  program  and  each  of  its  sub¬ 
routines  is  contained  in  Appendix  B.)  Each  of  the  main  subroutines 
manages  a  particular  display  associated  with  it.  We  will  discuss  the 
.operation  of  the  testing  program  in  terms  of  each  of  these  main  sub¬ 
programs.  Each  subroutine  has  the  same  logical  structure;  that  is, 
each  constructs  a  display,  then  waits  for  an  action  by  the  user  with 
the  light  pen.  The  routine  then  compares  the  location  of  the  light 
pen  with  various  figures  drawn  in  the  display  and  either  modifies  the 
display  to  reflect  the  most  recent  action  or  returns  control  to  the 


Fig.  8  —  State  diagram 
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main  program,  which  then  transfers  to  another  state.  In  general,  -khfcj  . 
user  changes  from  one  state  to  another  By  selecting  a  boxed  area  on  the 
face  of  the  display  with  his  light  pen.  These  areas  are  referred  to  as 
I light  buttons. 

j  The  main  program  serves  the  following  functions: 

•  It  collects  the  list  of  test  names  that  are  contained 
in  the  system  files. 

•  It  connects  the  program  with  the  terminal. 

•  It  asks  the  subject  to  submit  his  name. 

•  It  switches  among  the  main  subroutines. 

•  It  disconnects  the  program  from  the  terminal  when 
testing  is  complete. 

After  the  subject  has  typed  his  name  to  identify  himself  to  the 
system  (responding  to  the  display  "PLEASE  TYPE  YOUP,  NAME:"),  control 
is  passed  immediately  to  the  test  selection  subroutine. 

The  test  selection  subroutine  presents  the  user  with  the  display 
illustrated  in  Fig.  9.  The  two  boxes  in  the  lower  right-hand  corner 
of  Fig.  9  illustrate  the  light  buttons  mentioned  above.  If  the  user 
points  to  the  word  "QUIT,"  the  test  selection  subroutine  will  return 
to  the  main  routine  with  the  indication  that  the  main  routine  is  to 
vacate  the  operating  system  partition  and  the  computer.  If  the  user 
selects  the  "TEST"  light  button,  the  program  will  transfer  control  to 
the  test  administration  subroutine  if  a  test  has  already  been  selected. 
If  no  test  has  been  selected,  the  system  will  simply  remain  in  the  same 
state.  By  pointing  at  a  box  immediately  to  the  left  of  the  name  of  a 
test  (illustrated  on  the  left  half  of  Fig.  9),  the  user  indicates  that 
he  would  like  to  take  that  particular  test.  The  system  then  recovers 
that  test  information  from  the  computer  file  and  transfers  control  to 
the  test  administration  subroutine.  When  a  new  test  is  selected,  all 
of  the  information  normally  collected  by  the  test  administration  and 
data-processing  routines  is  reinitialized  so  that  no  information  is  j 
kept  between  test  administrations.  1 


a 

AVAILABLE  TESTS 


DIGEST  1 
DIGEST  2 
DIGEST  3 
DIGEST  4 
DIGEST  5 
DIGEST  6 
DIGEST  7 
DIGEST  8 
DIGEST  9 
DIGEST  10 
HUMANITIES  A 
HUMANITIES  B 
HUMANITIES  C 
HUMANITIES  D 
HUMANITIES  E 
HUMANITIES  F 
HUMANITIES  G 
NATURAL  SCIENCES  A 
NATURAL  SCIENCES  B 
NATURAL  SCIENCES  C 
NATURAL  SCIENCES  D 
SOCIAL  SCIENCES  A 
SOCIAL  SCIENCES  B 
SOCIAL  SCIENCES  C 
SOCIAL  SCIENCES  D 
SOCIAL  SCIENCES  E 
SOCIAL  SCIENCES  F 
SOCIAL  SCIENCES  G 


PLEASE  SELECT  A  TEST. 


Fig.  9 


Test  selection  display 


Figure  10  illustrates  one  of  the  two  displays  presented  by  the 
I  test  administration  subroutine.  The  test  used  in  the  figures  is  a 
'vocabulary  test.  The  word  to  be  defined  in  question  1  is  "AUSTERITY .  f* 

The  three  allowed  synonyms  are  "A:  EXTREME  SUFFERING,"  MB:  POVERTY," 
and  "C:  SEVERE  SIMPLICITY. n  For  any  given  question,  the  console  user 
may  select  a  point  within  the  triangle  labeled  A,  B,  C  to  indicate  the 
probabilities  he  associates  with  each  of  the  answers  labeled  A,  B,  C, 
in  the  upper  right-hand  corner  of  Fig.  10.  The  triangle  A,  B,  C  is  an 
equilateral  triangle  and  for  any  interior  point,  the  sum  of  the  dis¬ 
tances  from  that  point  to  each  of  the  sides  is  a  constant  and  can  be 
scaled  to  1.  In  this  way,  the  position  of  any  point  in  the  interior 
of  the  triangle  can  be  considered  to  be  a  selection  of  three  probabil¬ 
ities,  each  the  probability  of  correctness  associated  with  each  of  the 
answers  A,  B,  C.  When  the  user  selects  a  point  within  the  triangle, 
a  small  "x"  appears  at  that  point  (see  Fig.  10),  and  the  logarithmic 
scoring  system  is  evaluated,  using  the  three  probabilities.  The  upper 
left-hand  corner  display  in  Fig.  10  Indicates  what  scores  would  apply 
to  each  of  the  answers  A,  B,  C,  for  the  particular  response  represented 
by  the  x.  The  score  is  presented  in  two  ways:  one,  by  the  numeric  rep¬ 
resentation  of  the  score;  the  other  by  the  pictorial  representation.  The 
user  is  free  to  reselect  points  within  the  triangle  until  he  is  satis¬ 
fied  that  the  scores  displayed  in  the  upper  left-hand  corner  represent 
his  best  bet  for  each  of  the  possible  responses. 

Figure  11  illustrates  the  choice  of  option  C  with  full  certainty. 
jFigure  12  illustrates  the  situation  in  which  a  student  has  no  reason 
■to  choose  one  alternative  over  any  other.  The  scores  would  vary 
analogously  for  apex  A  or  for  apex  B,  When  he  is  satisfied  with  his 
selection,  he  may  do  one  of  several  things  as  represented  by  the  light 
buttons  in  the  lower  right-hand  corner  of  Fig.  10:  (1)  He  may  ask  for 

the  next  question;  (2)  he  may  ask  for  the  previous  question;  (3)  he  may 
ask  for  the  answer  to  his  question;  or  (4)  he  may  return  to  the  test 
selection  procedure. 

Whenever  he  selects  option  (3)  ,  he  will  be  presented  with  Fig.  13, 
which  illustrates  the  results  up  to  the  seventh  question.  The  box  in 
'the  upper_ left -hand corner  around  the  score  for  response  A  indicates 


A:  EXTREME  SUFFERING 


A  B:  POVERTY 


TEST 

SELECT 


QUEST  I  ON 
ANSWER 


PREVIOUS 

QUESTION 


NEXT 

QUESTION 


A  selected  point  and  associated  scores 


A:  EXTREME  SUFFERING 


B:  POVERTY 


PREV I OUa 
QUESTION 


A:  EXTREME  SUFFERING 


B:  POVERTY 


C 


TEST 

SELECT 


QUESTION 

ANSWER 


PREVIOUS 

QUESTION 


NEXT 

QUESTION 


No  preference  for  any  alternative 


Fig.  13  —  Cumulative  score  after  seventh  question 


which  of  the  three  answers  was  correct  and  the  numbers  below  indicatej 
how  many  points  were  gained  or  lost  as  a  result  of  the  userfs  assign- 
jment  of  probabilities.  The  triangle  has  been  replaced  by  a  rectangle! 
which  contains  a  plot  of.  the  cumulative  score  the  user  has  attained  as 

,he  has  gone  from  question  to  question.  The  line  sloped  to  the  upper 

I  -  : 

! right  is  the  line  he  would  follow  if  he  had  assigned  a  probability  ofi 
jl  to  each  of  the  correct  answers.  The  line  sloping  down  to  the  right 
is  the  line  he  would  follow  if  he  had  assigned  the  minimum  probability 
0.01  to  the  correct  answer  for  each  question.  Once  the  user  has  asked 
to  see  the  answer  to  a  question,  he  may  no  longer  change  the  proba¬ 
bilities  he  has  associated  with  the  answers.  That  is,  if  he  returns 
to  the  probability  assignment  display,  he  may  no  longer  select  a  new 
point  in  the  triangle.  If  he  attempts  to,  the  system  will  respond  with 
a  message  that  indicates  that  he  already  knows  the  answer  to  that 
question. 

If  the  user  selects  option  Cl)  (the  NEXT  QUESTION),  he  is  pre¬ 
sented  with  that  question  and  its  associated  responses.  If  the  user 
selects  option  (4)  CTEST  SELECT),  he  returns  to  the  test  selection  pro¬ 
gram  with  Fig.  13  displayed.  He  may  select  a  new  test,  thereby  de¬ 
stroying  the  information  he  has  collected  to  this  point.  Or  he  may 
simply  return  to  the  present  test  with  no  information  lost.  If  the 
..user  has  left  some  questions  unanswered  by  the  time  he  reaches  the  end 
of  the  test,  the  system  returns  to  the  unanswered  questions  in  turn 
until  each  question  has  been  answered.  When  the  final  question  has 
been  answered,  "TEST  SELECT"  changes  its  name  to  "SCORE  PAGE"  (see 
Fig.  1).  Figure  2  illustrates  the  answer  display  when  all  questions 
have  been  answered.  In  this  particular  case,  the  vertical  line  with 
"END"  above  it  coincides  with  the  right  boundary  of  the  plot.  If  there 
were  fewer  than  twenty  questions  in  the  test,  the  vertical  line  would 
appear  in  the  interior  of  the  plot.  If  the  user  now  selects  "SCORE 
PAGE,"  the  system  then  transfers  control  to  the  score  interpretation 
page. 

The  score  interpretation  subroutine  computes  the  variance  of  the; 
distribution  of  the  probabilities  and  the  realism  parameters,  that  is', 
the  slope  and  intercept  of  the  realism  line. _ It_then  computes  the _ j 


1 

perceived  and  actual  information  measures  and  constructs  the  upper  half 
of  the  display  illustrated  in  Fig,  4.  Actual  knowledge  is  plotted  frpm 
i  o  to  1  on  the  left  vertical  edge  and  perceived  knowledge  is  plotted  from 
10  to  1  on  the  right  vertical  edge  of  the  square.  The  line  joining  these 
two  points  helps  the  user  visualize  the  relationship  between  his  actual 
knowledge  and  what  he  has  perceived  his  knowledge  to  be.  If  the  vari¬ 
ance  of  the  probabilities  is  less  than  0.01,  the  system  assumes  that  it 
has  too  little  information  to  estimate  reliably  the  realism  line  and 
indicates  to  the  user  that  the  test  was  too  difficult  for  him.  If  this 
is  not  the  case,  the  system  then  computes  a  bias  measure  which  is  simply 
the  difference  between  the  perceived  information  measure  and  the  actual 
information  measure.  It  then  constructs  a  verbal  response  which  re¬ 
flects  this  bias.  Figure  14  indicates  decision  points  for  the  selec¬ 
tion  of  the  various  messages.  The  system  then  computes  the  original 
score  and  the  improved  score,  restates  them  in  light  of  the  number  of 
points  to  be  gained  rather  than  the  actual  number  of  points  attained, 
presents  them  as  part  of  messages  indicating  what  could  be  gained  Cl)  by 
improvement  in  realism,  (2)  by  gaining  more  information,  and  then  pre¬ 
sents  C3)  the  total  combined. 

After  observing  the  display,  the  user  may  Cl)  return  to  the  testing 
procedure,  C2)  return  to  the  test  selection  procedure,  C3)  request  a 
printed  summary  of  his  test  results  Csee  Fig.  15),  or  (4)  request  the 
realism  plot. 

Both  the  printed  summary  and  the  realism  plot  are  primarily  of  use 
to  the  investigator  rather  than  to  the  student.  The  summary  contains 
additional  information.  We  estimate  the  percentage  of  correct  answers 
a  student  would  have  obtained  under  the  classical  testing  method.  For 
each  question,  if  the  maximum  assigned  probability  corresponds  to  the 
correct  answer,  we  count  that  answer  as  correct.  If  the  student  assigns 
a  maximum  probability  to  several  responses,  including  the  correct  one, 
we  cannot  treat  the  situation  directly,  so  we  treat  it  probabilistically 
and  assign  a  value  of  to  this  question,  where  E  is  the  number  of  times 
jthe  maximum  probability  is  used  in  that  question.  The  average  value  of 
j  these  assignments  over  all  questions  represents  an  estimate  of  what  tjie 
student  would  have  obtained  had  he  been  restricted  to  discrete  choicek. 
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©)  =  "  Overvalue  " 


"The  way  you  placed  your  bets  on  this  test 
shows  you  are  very  accurate  in  assessing 
the  validity  of  your  facts  and  reasons" 


®  =  "Undervalue" 


(A)  =  "Considerably  " 

©  =  "Much  less" 

0.7 

(A)  =  Null 
©  =  "Less" 

0.4 

© 

0.1 

"You  tend  to  (A)  your  knowledge" 

"  Bet  (C)  heavily  on  the  answers  you 
feel  are'  correct" 

-0.1 

(J)  =  "  Somewhat  " 

©  =  "More" 


®  =  Null 
©  =  "More " 

-0.7 

{ 2)  =  "Considerable" 
©  =  "Much  more  " 

-1.0 


=  "  Somewhat 11 
=  "Less" 


Fig  .  14  —  Bias  decision  points 
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Fig.  15  —  Printed  test  summary 


I  This  proportion  of  correct  answers  can  be  considered  a  base  value  Z  j 

’and  a  base  line  drawn  through  the  points  (1/3,  1/3)  and  (1,3).  We  j 

i  - -  •  ! 

also  develop  a  STEREOTYPE  score  by  using  the  probability  Z  if  the  corr 

:rect  .answer  were  assigned  the  maximum  probability,  (l-Z)/2  otherwise. 
Finally,  we  compute  a  BIAS  indicator  which  is  simply  the  perceived  in¬ 
formation  less  the  revised  information.  The  asterisks  indicate  the 
correct  answer  in  the  columns  under  "PROBABILITIES"  and  "SCORES." 

The  realism  plot  subroutine  constructs  a  display  of  the  external 
validity  graph  with  the  associated  plotted  lines  (Fig.  16).  The  verti¬ 
cal  lines  in  the  display  are  a  histogram  of  the  frequency  of  use  of 
each  of  the  probabilities  along  the  bottom  of  the  diagram.  The  class 
interval  for  this  histogram  is  0.1.  The  xTs  distributed  throughout 
the  display  are  the  relative  frequencies  with  which  elements  in  those 
class  intervals  were  associated  with  the  correct  answer.  The  solid 
line  going  from  the  lower  left  to  the  upper  right-hand  corner  of  the 
display  is  the  ideal  realism  line.  The  solid  line  at  an  angle  to  the 
ideal  line  is  the  fitted  realism  line  and  the  dashed  line  is  the  base 
line.  From  this  point  the  user  may  return  to  the  score  page  or  the 
test  procedure. 

Since  this  version  of  the  measurement  program  is  primarily  an  ex¬ 
ploratory  one,  the  software  was  designed  to  make  that  exploration  as 
easy  as  possible.  The  various  functions  of  test  selection,  test  admin¬ 
istration,  and  the  two  feedback  processes  are  all  isolated  to  allow 
changes  to  be  made  in  each  without  affecting  the  others.  The  logic 
in  each  one  of  the  major  subroutines  is  as  simple  as  possible  to  allow 
ease  of  change. 


‘V 
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III,  PLANNED  MODIFICATIONS  AND  EXTENSIONS 


•PHILOSOPHY  OF  USE  j 

The  general  philosophy  of  the  present  form  of  the  testing  program 
|  is  that  everything  should  be  as  simple  and  as  easy  to  use  as  possible;, 
j The  user  should  not  be  concerned  about  the  fact  that  he  is  using  the  j 
computer  and  that  fact  should  not  overawe  him.  Aside  from  considera¬ 
tions  of  the  responsiveness  and  stability  of  the  system,  a  major  con¬ 
cern  is  with  the  ability  of  the  student  to  grasp  the  relation  between 
his  score  and  his  actions  with  the  triangle.  He  must  be  allowed  a 
maximum  amount  of  flexibility  and  the  system  should  not  encourage  him 
to  fall  into  the  use  of  stereotyped  responses.  In  addition,  the  system 
should  be  as  nearly  self-teaching  as  possible.  That  is,  the  actions 
to  be  taken  should  be  as  self-evident  as  they  can  be  made.  The  extent 
to  which  the  latter  situation  can  be  attained  is  now  under  investiga¬ 
tion  by  observing,  questioning,  and  recording  the  responses  of  users 
of  the  test  procedure. 

POSSIBLE  MODIFICATIONS 

Since  textual  messages  are  difficult  to  construct  and  are  some¬ 
times  misleading  (e.g. ,  the  bias  and  bet  messages),  it  seems  best  to  ; 
present  the  information  to  the  student  in  some  nonverbal  form.  One 
possible  approach  to  such  a  presentation  would  be  to  eliminate  a  good 
part  of  the  display  on  the  score  page  and  to  allow  the  student  to  re¬ 
cycle  through  the  test,  having  the  system  present  in  the  triangle  both 
his  original  response  and  the  response  computed  by  the  system  on  the 
basis  of  the  realism  line  (Fig.  16).  This  graphical  display  would 
illustrate  to  the  student  how  he  should  modify  his  behavior  to  maximize 
his  score  with  his  present  level  of  knowledge.  The  appropriate  kind 
and  amount  of  feedback  and  advice  to  be  provided  the  student  will  be  j 
the  major  focus  of  our  research  in  the  immediate  future. 

;  The  present  system  has  no  provision  for  permanently  collecting  j 

'data  in  machine  readable  form  on  the  student fs  responses.  As  we  gain| 
experience  with  the  data  represented  by  Fig.  17,  we  will  design  and _ 1 
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Fig.  17 —  Adjusted  response 
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Implement  a  scheme  for  managing  the  collection  of  data  over  many  stu-j 
dents  and  many  tests  and  for  performing  appropriate  statistical  analyses 
of  the  results.  In  fact,  an  important  goal  of  our  work  is  to  provide; 
the  ultimate  user  of  admissible  probability  measurement  with  a  com¬ 
plete  set  of  tools  for  data  collection,  analysis,  and  interpretation. 


IV.  PLATO  IV 


The  PLATO  IV  system  is  designed  to  be  an  economically  viable  large- 
scale,  computer-based  education  system. The  computer-based  Education 
Research  Laboratory  at  the  University  of  Illinois  has  developed  versions 
of  the  system  from  a  single  terminal  connected  to  a  1954  vintage  computer 
to  a  collection  of  20  graphic-pictorial  terminals  connected  to  a  more: 
powerful  computer.  Over  the  period  of  development  the  laboratory  has 
gathered  data  on  more  than  70,000  student  contact  hours.  The  present 
version  (PLATO  IV)  is  designed  to  bring  the  terminal /computer  costs  to 
within  the  25q  to  30C  range  per  student  contact  hour. 

A  PLATO  IV  terminal  is  an  interactive  computer  graphics  terminal 

ro\ 

with  the  following  features: 

1.  An  8*3f  inch  square  plasma  display  panel  that  is  readable  in  a 
brightly  lighted  room  without  eyestrain. 

2.  Permanent  storage  of  information  on  the  display  screen  with¬ 
out  flicker.  Absolutely  no  refreshing  of  the  display  panel 
by  the  computer  is  required.  Selective  "WRITE  and  ERASE  capa¬ 
bility  exists  for  all  dots,  lines,  and  characters. 

3.  Self-contained  character  and  line  generators. 

4.  A  character  writing  speed  of  180  characters  per  second  and 
the  capability  of  displaying  2048  characters  on  the  screen. 

5.  A  line  drawing  speed  in  excess  of  600  inches  per  second. 

6.  A  character  repertoire  of  252  characters,  126  of  which  are 
alterable  through  the  computer  program. 

7.  The  ability  to  transmit  and  receive  data  on  voice  trade  tele¬ 
phone  circuits. 

8.  A  random-access  slide  projector  for  rear  projection  of-  static 
information  on  the  display  screen. 

9.  Additional  input-output  channels  for  the  control  of  auxiliary 
equipment . 

10.  An  optional  random-access  audio  response  unit.  ( 
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11.  A  pointing  matrix  for  the  surface  of  the  display.  An  inter-J 

■ .  _  -  i 

ruption  of  light  beams  from  two  edges  of  the  display  to  the  j 
opposite  edges  determines  a  point  in  the  matrix.  That  point 
is  reported  to  the  computer  and  correlated  with  the  contents 
of  the  display.  The  matrix  has  16  divisions  in  each  dimension 
of  an  8-J  x  8-1  inch  area. 

;  PLATO  IV  is  designed  to  support  remote  classrooms  consisting  of 

jup  to  32  terminals.  The  data  rate  to  each  terminal  varies  from  2§  to 
80  10-bit  characters  a  second,  depending  on  the  activity  of  the  32 
terminals. 

The  PLATO  TV  hardware  and  software  are  designed  specifically  to 

deal  simultaneously  with  large  numbers  of  students,  each  following  his 

own  course  of  study.  A  course  author  Cin  our  case,  a  test  author)  writes 

(9) 

his  course  material  in  the  TUTOR  language. v  '  TUTOR  allows  the  author 
to  structure  his  material  easily  to  accommodate  a  wide  variety  of  stu¬ 
dent  responses.  In  addition,  the  language  includes  facilities  for  data 
collection  and  computation  as  well  as  access  to  files  of  course  materials. 

The  capabilities  of  the  PLATO  IV  system  and  the  availability  of 
terminals  in  the  military  services  provide  a  vehicle  for  extensive  test¬ 
ing  of  the  admissible  probability  measurement  concept  in  the  military 
environment.  Our  intent  is  to  develop  a  PLATO  IV  version  of  admissible 
probability  measurement,  including  appropriate  data  collection,  analysis, 
and  interpretation  as  a  package  for  the  military  services.  We  hope  to 
encourage  and  assist  the  services  to  incorporate  computer-based  admis¬ 
sible  probability  measurement  in  their  education  and  training  programs. 
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j  SUPPORT  SOFTWARE  j 

VPACK  ROUTINES 

The  VPACK  display  routines  operate  on  a  buffer  provided  by  the 
calling  program.  The  programmer  specifies  the  address  and  size  of  the 
buffer  during  initialization.  Thereafter,  VPACK  retains  this  informal 
tion  so  that  the  programmer  no  longer  need  be  concerned  with  it.  VPACK 
also  maintains  information  about  the  amount  of  buffer  space  currently 
being  used  (the  current  length}  and  the  position  into  which  the  display 
commands  are  being  stored  (the  current  position).  These  data  are  auto¬ 
matically  updated  by  the  several  routines  as  appropriate.  (See  Fig. 

18.)  The  current  length  of  a  buffer  may  exceed  the  maximum  length 
allowed  by  the  Video  Graphic  System  (VGS)  for  a  single  picture  segment. 
In  this  event,  VPACK  automatically  segments  the  picture  at  appropriate 
points  and  passes  the  multiple  segments  to  VGS. 

More  detailed  descriptions  of  the  subroutines  follow,  in  alpha¬ 
betical  order. 

VCHAR:  Adds  a  character  string  to  the  buffer. 

CALL  VCHAR  (IX,  IY,  ST,  N) 

VCHAR  adds  N  characters  from  the  string  ST  to  the  buffer,  begin¬ 
ning  at  the  current  position.  IX  and  IY  specify  the  center  of  the  first 
character  position.  The  intensity  of  the  string  is  set  to  normal.  The 
current  position  and  current  length  are  updated. 

VCOPY:  Produces  hardcopy  of  the  current  picture. 

CALL  VCOPY  (L) 

VCOPY  effectively  produces  hardcopy  (both  paper  and  35mm  film)  of 
| the  current  picture.  More  specifically,  it  reformats  the  buffer  intoj 
| a  meta  language,  which  it  writes  on  a  disc  data  set  named  SC4060ZZ.  j 
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Fig.  18  —  Use  of  current  position  and  current  length 
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This  data  set  is  subsequently  transferred  to  magnetic  tape  and  pro-  , 
cessed  off-line,  ; 

Parameter  L  specifies  the  action  to  be  taken  by  the  microfilm 
processor  after  drawing  the  image.  Its  values  are: 

0  Produce  11  x  14  copy  and  advance  the  frame 

1  Produce  11  x  14  copy  but  do  not  advance  the  frame 

2  Produce  8  x  H  copy  and  advance  the  frame 

3  Produce  8  x  11  copy  but  do  not  advance  the  frame 

4  Close  the  output  file 

In  normal  operation,  the  frame  is  advanced  after  each  frame  is  drawn. 

If  the  frame  is  not  advanced,  the  next  picture  that  is  copied  will  be 
superimposed  onto  the  current  picture. 

VINIT :  Initializes  the  display,  tablet,  and  keyboard. 

CALL  VINIT  (NAME, BUFF, N) 

VINIT  connects  the  program  to  the  VGS  and  activates  the  tablet  and 
keyboard.  NAME  is  a  string  of  exactly  eight  characters  which  the  user 
will  type  to  connect  his  terminal  to  the  program  (CREATE  NAME)  .  If 
the  name  consists  of  fewer  than  eight  characters,  it  must  be  padded 
with  trailing  blanks.  BUFF  is  the  buffer,  which  VINIT  initializes  with 
a  RESET  command.  N  is  the  maximum  length  of  BUFF  in  bytes.  The  current 
position  and  current  length  are  initialized. 

An  appropriate  value  for  N  can  be  estimated  by  considering  the 
longest  buffer  that  will  be  created,  using  the  following  guide: 

•  Each  call  to  VCHAR  requires  n  +  6  bytes,  where  n  is  the 
length  of  the  character  string. 

•  Each  call  to  VLINES  requires  10  n  +  1  bytes,  where  n  is 
the  number  of  discrete  line  segments. 

•  Each  call  to  VJLINE  requires  5  n  +  6  bytes ,  where  n  is 
the  number  of  joined  line  segments. 


j  •  Each  call  to  VNULL  requires  n  bytes ,  where  n  is  the 

;  number  of  null  codes  to  be  added, 

•  Each  call  to  VSIZE,  VLTYPE,  or  VDIM  requires  1  byte. 

•  All  other  calls  require  no  space. 

VJLINE :  Adds  joined  line  segments  to  the  buffer. 

CALL  VJLINE  (N,IX,IY) 

VJLINE  adds  N  joined  line  segments  to  the  buffer,  beginning  at  the 
current  position.  The  lines  are  set  to  normal  intensity.  The  current 
position  and  current  length  are  updated. 

The  1th  line  extends  from  the  point  IX (1),  IY(1)  to  the  point 
IX (1+1) ,  IY(1+1).  Arrays  IX  and  IY  must  therefore  contain  at  least 
N  +  1  entries.  All  coordinates  after  the  first  are  entered  in  the 
buffer  relative  to  the  first;  this  enables  the  program  to  move  the 
entire  set  of  lines  by  a  single  call  to  VMOVE. 

VLINES :  Adds  discrete  line  segments  to  the  buffer, 

CALL  VLINES  (N , 1XF , IYF , IXT , IYT ) 

VLINES  adds  N  discrete  line  segments  to  the  buffer,  beginning  at 
the  current  position.  The  line  segments  are  set  to  normal  intensity. 

The  current  position  and  current  length  are  updated. 

The  1th  line  extends  from  IXF(T),  TYF(I)  to  IXT(I),  IYT(I).  All 
coordinates  after  the  first  are  entered  in  the  buffer  relative  to  the 
first;  this  enables  the  program  to  move  the  entire  set  of  lines  by  a 
single  call  to  VMOVE. 

VLTYPE:  Adds  a  line-type  code  to  the  buffer, 

CALL  VLTYPE  (L) 

VLTYPE  adds  a  line-type  code  to  the  buffer  at  the  current  posi¬ 
tion.  The  current  position  and  current  length  are  updated.  The  line- 
type  code  is  a  model  relative  to  the  buffer;  that  is,  once  the  line- 
type  has  been  specified,  it  applies  to  all  lines  appearing  in  the  buffer 
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until  the  next  line-type  code  appears.  If  no  line-type  code  is  speci¬ 
fied,  solid  lines  are  used.  The  values  of  L  are: 

1 .  Solid  lines 

2.  Dashed  lines 

3.  End  points  only 

VORG :  Resets  the  current  buffer  position. 

CALL  VORG  (J) 

VORG  sets  the  current  position  to  the  value  of  J.  The  current 
length  is  unchanged.  The  value  of  J  must  not  exceed  the  current  length. 
This  routine  allows  the  programmer  to  reconstruct  part  of  the  display 
by  replacing  a  sequence  of  buffer  orders  with  another  sequence  of  the 
same  length.  To  guarantee  that  identical  lengths  are  used,  it  is  good 
practice  to  replace  character  strings  only  with  other  character  strings, 
sequences  of  joined  lines  with  other  sequences  of  joined  lines,  etc., 
and  to  ensure  that  the  number  of  characters  or  lines  is  the  same  as 
the  number  being  replaced, 

VLPEN:  Communicates  with  the  light  pen. 

CALL  VLPEN  ClACT ,  ISIZE ,  ITIME ,  IDATA,  IEXIT) 

VLPEN  is  an  adaptation  of  a  routine  which  communicates  with  the 
Rand  Tablet  and  is  used  for  handprinted  symbol  recognition  studies. 

The  parameters  of  interest  for  the  light  pen  are  IDATA  and  IEXIT. 

IDATA  is  a  two-entry  integer  array  which  contains  the  (x,y)  location 
of  a  light  pen  strike.  IEXIT  is  an  integer  which  indicates  the  reason 
for  the  return  from  the  routine.  VLPEN  normally  waits  for  a  light  pen 
interrupt  before  returning  control  to  the  calling  routine. 

VPOSIT :  Retrieves  the  current  buffer  position. 


CALL  VPOSIT  (J) 
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VPOSIT  returns  the  current  position  in  location  J.  The  current  i 

i  ; 

position  and  current  length  are  unchanged, 

i 

VSEND:  Sends  a  picture  to  the  screen. 

CALL  VSEND 

VSEND  sends  an  updated  picture  to  the  screen.  Because  all  of  the 
other  picture  manipulation  routines  merely  alter  the  buffer,  this 
routine  must  be  called  to  make  their  effects  visible  to  the  user.  The 
current  position  and  current  length  are  unchanged. 

VTRUNC :  Truncates  the  buffer. 

CALL  VTRUNC  (J) 

VTRUNC  sets  the  current  length  to  the  value  of  J.  If  the  current 
position  exceeds  the  current  length.  This  routine  effectively  deletes 
all  display  code  appearing  in  the  buffer  beyond  the  specified  point. 

FORTRAN  CONVERSION  ROUTINES 


FORMAT:  Converts  a  number  to  a  character  string. 

CALL  FORMAT  ClFMT ,  IN,  ID , VALUE,  CHARS) 

FORMAT  converts  a  number,  either  real  or  integer,  to  a  character 
string  in  I,  F,  or  E  format.  IFMT  specifies  the  type  of  conversion  to 
be  performed:  (1)  to  convert  an  integer  to  I  format;  (2)  to  convert 
a  real  number  to  F  format;  or  (3)  to  convert  a  real  number  to  E  format. 
IW  is  the  field  width  of  the  converted  number.  ID  specifies  the  number 
of  decimal  places;  ID  =  0  if  IFMT  =  1.  VALUE  contains  the  number  to 
convert;  it  may  be  either  real  or  integer,  depending  on  the  value  of 
IFMT.  CHARS  will  contain  the  resulting  character  string;  it  must  be 
an  array  large  enough  to  hold  at  least  IW  characters. 


CONVRT:  Converts  a  character  string  to  a  number. 

CALL  CONVRT  Cl FMT,  IW,  STRING, OUT) 

CONVRT  converts  IW  characters  of  the  character  string  STRING  to 
a  number,  either  real  or  integer*  IFMT  specifies  the  conversion: 

(1)  to  convert  to  an  integer;  C2)  to  convert  to  a  real;  or  (3)  to  con¬ 
vert  to  a  real  if  a  decimal  point  or  the  letter  E  appears  in  the  string, 
but  to  an  integer  otherwise.  On  exit,  OUT  contains  the  converted  value, 
either  real  or  integer. 
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~  Append ix~B - 

PROGRAM  LISTING 


MAIN  PROGRAM 


C  ****  ARRAYS  **** 

DIMENSION  Ml (7) 

DIMENSION  QUEST ( 16,4,40) 

DIMENSION  CHOICE(8,3,3,40),ANSW(40) 

DIMENSION  RESPON( 3, 40) , LOCKED (40) 

DIMENSION  TEST(5),SUBJCT(16) 

DIMENSION  IDATA( 2) 

DIMENSION  BUFF ( 1200) .MARK ( 20 ) , BLANKS ( 1 6) 

DIMENSION  FIL£S(5,50) 

C  ****  TYPE  DECLARATIONS  **** 

INTEGER  Ml , QUEST, ANSW, CHOICE, TEST, SUBJCT, FILES 
INTEGER  SAVE, SCORE, LOCKED 
q  ***☆  DATA 

DATA  Ml /'PLEA', 'SE  T', 'YPE  ','YOUR','  NAM', 'E  ','  7 

DATA  BLANKS/ 1 6* '  '/ 

DATA  MDIM/40/ 

C  ***************  FORMAT  STATEMENTS  *************************** 
100  FORMAT (14) 

200  FORMAT ( 1 6A4) 

C  ************  read  NAMES  OF  AVAILABLE  TESTS  ***************** 
READ  (9,100)  NFILES 
DO  300  1=1, NFILES 

300  READ  (9,200)  (F I LES (K, I ) ,K= 1 , 5 ) 

REWIND  9 

C  *********■***£  INITIALIZE  *********** ************** *********** 

I NDEX=0 

C  ************  CONNECT  TO  THE  TERMINAL  *********************** 
CALL  VLINIT( 'TEST  '.BUFF, 4700) 

CALL  VPOSIT (  MARK(l)) 

CALL  VEDSET(0, 0,2,0) 

C  ****  GET  THE  SUBJCT'S  NAME  **** 

CALL  VORG(MARK( 1 )) 

CALL  VCHAR( 0, 500.M1 ,28) 

CALL  VPOSIT ( ITERM) 

CALL  VTRUNC(  ITERM) 

CALL  VSEND 

CALL  VEDIT(0,450, 64, BLANKS,  SUBJCT,  I  TERM) 

C  **********  SELECT  THE  TEST  ************************** 

400  CALL  SELECT(MDIM,INDEX,MARK, NFILES, FILES, QUEST, CHOICE, ANSW, 
X  RESPON, LOCKED, BLANKS, TEST, N, NEXT ) 

GO  TO  (400.500,600,700,800),  NEXT 
C  **********  QO  THE  TESTING  ************************** 

500  CALL  TESTER(MDIM, MARK, QUEST, CHOICE, ANSW, RESPON, LOCKED, 

X  INDEX, N, BLANKS, NEXT) 

GO  TO  (400,500,600,700,800),  NEXT 
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C  **************  PRINT  THE  RESULTS  ******************** 

600  CALL  PRINKMOIM,  INDEX, NyRESPON, ANSW, MARK, BLANKS, TEST^SUBJCT, 
X  GEOM,PVAR, SCORE, SREAL, TREAL,SBASE , TBASE, NEXT) 

GO  TO  (400,500,600,700,800),  NEXT 
q  *vr ********  PLOT  THE  RESULTS  ************************** 

700  CALL  PLOT(MDIM,MARK,SCORE,GEOM, TEST, SUBJCT, SREAL, TREAL, 

X  SBASE, TBASE, N,RESPON, ANSW, NEXT) 

GO  TO  (400,500,600,700,800),  NEXT 
q  ************  QUIT  QUIT  QUIT  ************************** 

800  CALL  VTRUNC (MARK( 1 ) ) 

CALL  VSEND 
CALL  EXIT 

Q  ***********************  *  *******  *  W*  ****************  ***** 

ENO 
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TEST  SELECTION  SUBROUTINE 


SUBROUTINE  SELECT(/MDIM/, /INDEX/, /MARK/, /NFILES/ ,/FILES/ , 

/  ai  ir  /  J  r'  \  \  t  r'  r~  t  /  A  iuC  IJ  /  /DCCDHM/ 


DIMENSION 

DIMENSION 

DIMENSION 

Q  ************* 

DIMENSION 

DIMENSION 

DIMENSION 

Q  ************* 


X 

X 

[ 

X 

X 


x  /QUEST/, /CHOICE/, /ANSW/./RESPON/, 

X  /LOCKED/, /BLANKS/, /TEST/, /N/, /NEXT/) 

MARK( 1 ) ,FILES(5,MDIM) , QUEST ( 1 6, A,MDIM) 
CHOICE(8,3,3,MDIM)  ,  % 

ANSW( 1 ) ,BLANKS( 1 ),TEST( 1 ) ,RESPON(3,MDIM), LOCKED! 1 ) 
DIMENSION  STATEMENTS  ************************* 

M2  ( 6) 

IDATA( 2) 

BOXX(5,5O),BOXY(5,50) 

w  ^  ^  ^ _ TYPE  DEC  LAP  AT  IONS  **************************** 

INTEGER  MARK, FI LES, QUEST, CHOICE, ANSW, LOCKED, BLANKS, TEST 
INTEGER  M2 

INTEGER  IDATA,BOXX,BOXY 

**************  DATA  STATEMENTS  ***************************** 
DATA  M2/'PLEA',  'SE  S', 'ELEC', 'T  A  '.'TEST','.  7 

DATA  BOXX/ 875, 975, 9 75, 875, 875, 

875,975,975,875,875, 

0,10,10,0,0,235*0/ 

DATA  BOXY/ 10,10,85,85, 10, 

110, 110,185,185, 110, 

990, 990,999,999,990,235*0/ 

DATA  NARRAY/50/ 

c  ***************  FORMAT  STATEMENTS  *************************** 
100  FORMAT ( 1 4 ) 

200  FORMAT ( 1 6A4)  *,,***..* 

c  ************  START  BY  GETTING  THE  TEST  NAME  ***************** 

CALL  VORG(MARK( 1 )) 

CALL  VCHAR( 75,950,  'AVAILABLE  TESTS  ,15) 

DO  300  KK=1 , NFILES 

CALL  VCHAR( 75, 950- ( KK+ 1 )*24,FILES(1 ,KK),20) 

BOXX( 1 ,KK+3 )=0 

B0XX(2,KK+3)=50 

B0XX(3,KK+3)=50 

BOXX(4,KK+3)=0 

B0XX(5,KK+3)=0 

BOXY ( 1 ,KK+3 )=95Q- (KK+1 )*24- 1 0 
B0XY(2,KK+3)=B0XY( 1 , KK+3) 

BOXY ( 3 , KK+3 )=BOXY ( 1 , KK+3 )+20 
BOXY( 4 ,KK+3 ) =BOXY( 1 , KK+3 ) +20 
BOXY ( 5,KK+3 )=BOXY ( 1 , KK+3 ) 

CALL  VJLINE(4,B0XX(1, KK+3), B0XY(1, KK+3)) 

300  CONTINUE 

CALL  VJLINE(4,B0XX(1,1),B0XY(1,1)) 

CALL  VCHAR(B0XX( 1 , 1 ) +24, BOXY (1,1 )+36,  TEST  ,4) 
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CALL  VJLINE(4,B0XX<1,2),B0XY(1,2)) 

CALL  VCHAR(B0XX(1,2)+24,B0XY(1,2)+36,  QUIT  ,4) 

CALL  VCHAR(75,950-(NFILES+3)*24,M2,24) 

CALL  VJLINE(4,B0XX( 1 , 3) , BOXY( 1 , 3  ) ) 

CALL  VPOSIT ( ITERM) 

CALL  VTRUNC ( I  TERM) 

CALL  VSEND 

C  **********  WAIT  FOR  PEN  ACTION  ***************************** 
400  CALL  PEN( IDATA, BOXX, BOXY,NARRAY,NBOX) 

IF ( NBOX.EQ.O)  GO  TO  400 
IF(NB0X.GT.3)  GO  TO  600 
IF ( NB0X.EQ.2 )  GO  TO  500 
IF(NB0X.EQ.3)  GO  TO  1500 

Q  ***********  RETURN  TO  TESTING  ****************************** 
IF( INDEX. EQ.O)  GO  TO  400 
NEXT=2 
RETURN 

q  ************  QUIT  ****************************************** 
500  NEXT=5 
RETURN 

c  *****  RESET  INTERNAL  RECORDS  ***** 

600  I ND E X= 1 

DO  700  1-1,3 
DO  700  J= 1 ,MDIM 
700  RESPON! I, J)=0. 

DO  800  I=1,MDIM 
800  LOCKED! I )=0 

C  *****  INPUT  TEST  QUESTIONS  AND  ANSWERS  ***** 

DO  900  1=1,5 

900  TEST(I)=FILES(I,NB0X-3) 

CALL  VORG(MARK( 1  )) 

CALL  VCHAR( 5 00, 500, TEST, 20) 

CALL  VPOSIT! ITERM) 

CALL  VTRUNC! ITERM) 

CALL  VSEND 
KK=NB0X+6 
READ  ( KK, 100)  N 
DO  1400  1=1, N 
DO  1 000  J=1 ,4 

1000  READ  (KK, 200)  ( QUEST (K,J,I),K=1,16) 

DO  1100  J= 1 ,3 
DO  1100  L=1 ,3 

1100  READ  (KK, 200)  (CHOICE(K,L, J, I ) ,K=1 , 8) 

CALL  VORG(MARK( 1 ) ) 

KKK=MOD( 1,2) 

IF  (KKK.NE.O)  GO  TO  1200 
CALL  VCHAR( 500, 500, TEST, 20) 

GO  TO  1300 

1200  CALL  VCHAR(500, 500, BLANKS, 20) 

1300  CALL  VPOSIT! ITERM) 
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CALL  VTRUNC( I  TERM) 
CALL  VSEND 

1400  READ  (KK, 1 00)  ANSW(I) 
REWIND  KK 
NEXT=2 
RETURN 

1500  CALL  VCOPY ( 2 ) 

GO  TO  400 
END 
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TEST  ADMINISTRATION  SUBROUTINE 


SUBROUTINE  TESTER(/MDIM/, /MARK/, /QUEST/, /CHOICE/, /ANSW/, 
X  /RESPON/, /LOCKED/,/ INDEX/, /N/, /BLANKS/, /NEXT/) 

DIMENSION  MARK( 1 ), QUEST ( 1 6,4, MDIM) , CHOICE (8,3,3, MDIM), 

X  ANSW( 1 ) ,RESPON( 3,MD IM) , LOCKED ( 1 ) , BLANKS ( 1 ) 

C  *********************  DIMENSIONS  ************************** 
DIMENSION  BO XX (5, 5), BOXY (5,5) 

DIMENSION  TRI X( 4) , TRI Y( 5) 

DIMENSION  I X ( 5 1 ), I Y ( 5 1 ) 

DIMENSION  PTS(5) ,SCR(3) ,M3(5) 

DIMENSION  IDATA( 2) 

DIMENSION  MMAX( 14),MMIN( 1 4 ) , THEANS ( 9 ) 
q  *******************  TYPE  DECLARATIONS  ********************** 
INTEGER  MARK, QUEST, CHOICE, ANSW, LOCKED, BLANK, SCORE 
INTEGER  BOXX,BOXY 
INTEGER  TRIX,TRI Y 

INTEGER  R1X,R1Y,R2X,R2Y,R3X,R3Y, QPOSX, QPOSY 
INTEGER  PTS, SCR, M3, REVIE, THEANS 
INTEGER  IDATAY, NARRAY 

q  *******************  INITIAL  DATA  ***********************^*** 
DATA  TRIX/425,50,800,425/ 

DATA  TRIY/700,50,50,700/ 

DATA  BOXX/875, 995,995,875,875, 

X  875,995,995,875,875, 

X  875,995,995,875,875, 

X  875,995,995,875,875, 

X  0,10,10,0,0/ 

DATA  BOXY/ 10,10,85,85,10, 

X  110,110,185,185,110, 

X  210,210,285,285,210, 

X  310,310,385,385,310, 

X  990,990,999,999,990/ 

DATA  QPOSX, QP0SY,R1 X, R2X,R3X,R1 Y,R2Y,R3Y 
X  /95, 975, 525, 525, 525, 825, 725,625/ 

DATA  PTS/'  V.. 

DATA  M3/'QUES',  'TION',  '  %  P','LEAS  ,  E  / 

DATA  NARRAY/5/ ,MAXPLT/20/  . 


%'I 

'0 


DATA  MMAX/'M  ' ,  'A  ','X  ',  'I  'M  ','U 

i  x  y  y?  vo  vi  vn  v 

XDATA  MMIN/'M  'I  ',  'N  ',  'I  'M  ','U  ', 

X  'mV  ,  P  ,  0  ,  I  ,  N 

X  'j  '  's  '  j 

DATA  THEANS/'SQRR', 'Y,  Y','0U  A 'LREA 'DY  K','NOW  ', 
X  'THE  ','ANSWVER.  V 

£  **********  THE  SCORING  FUNCTION  ***********************  j 


','1 

'0 


Vm 

Vi 

','M 

','1 


','U 


','U 
'.  'N 
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VALUE(X)=50.*AL0G10(3.*AMAX1 (.01 , X) ) 
q  *****************  DRAW  THE  BOXES  *************************** 

'  CALL  VORG(MARK( 1 ) )  i 

C  ****  DRAW  THE  'NEXT'  BOX  **** 

CALL  VJLINE(4,B0XX(1,1),B0XY(1,1 )) 

CALL  VCHAR( BOXX( 1,1)+36,B0XY(1,1)+60, 'NEXT', 4 ) 

CALL  VCHAR(BOXX(1,1)+12,BOXY(l,l)+36, 'QUESTION', 8) 

C  ****  DRAW  THE  'PREVIOUS'  BOX  **** 

CALL  VJLINE(4,B0XX(1,2), BOXY (1,2)) 

CALL  VCHAR(B0XX(1,2)  +  12,B0XY(1,2)+60, 'PREVIOUS', 8) 

CALL  VCHAR(BOXX( 1 , 2)  + 1 2, BOXY( 1 ,2)+36,  'QUESTION ', 8) 

Q  ****  DRAW  THE  'ANSWER'  BOX  **** 

CALL  VJLINE(4,BQXX(1,3), BOXY (1,3)) 

CALL  VCHAR(BOXX( 1,3)+12,BOXY( 1 ,3 )+60, 'QUESTION ',8) 

CALL  VCHAR(  BOXX(  1 ,3 )+24, BOXY (  1 ,3  )+36,  'ANSWER  ',6) 

0  ****  DRAW  THE  'SELECT'  BOX  **** 

CALL  VJLINE(4,BOXX(l,4),BOXY(1,4)) 

CALL  VJLINE(4,B0XX(1,5), BOXY (1,5)) 

CALL  VPOS I T ( MARK ( 2 ) ) 

C  ****  GET  THE  QUESTIONS  AND  CHOICES  **** 

100  CALL  VORG(MARK( 2 ) ) 

CALL  VPOS I T ( I  TERN) 

CALL  VTRUNC ( I  TERM) 

DO  200  1=1,4 

:  200  CALL  VCHAR( QP OS X, QPOSY- ( I- 1 ) * 24 , QUEST ( 1 ,1,1 NDEX) ,  64 ) 

CALL  VPOS I T (NARK ( 3 ) ) 

DO  300  1=1,3 

:  300  CALL  VCHAR(R 1 X,R1 Y-( 1-1 ) *24, CHOICE ( 1 , I , 1 , INDEX) ,32) 

CALL  VPOS IT (MARK(4) ) 

DO  400  1=1,3 

400  CALL  VCHAR(R2X,R2Y-( 1-1 )*24,CH0ICE( 1 , 1,2, INDEX) ,32) 

CALL  VPOS IT (NARK (5 ) ) 

DO  500  1=1,3 

500  CALL  VCHAR( R3X, R3 Y- (I-1)*24, CHOICE (1,1, 3, INDEX), 32) 

CALL  VPOS I T (MARK( 6) ) 

C  ****  DRAW  RESPONSE  DIAGRAM  OUTLINE  **** 

600  IF ( ( RESPON( 1 , INDEX) .EQ.O. ). AND. (RESP0N(2, INDEX) . EQ. 0 .) .AND. 
X  (RESP0N(3, INDEX). EQ.O. ))  GO  TO  1000 
C  ****  ip  no  RESPONSE  SKIP  TO  DRAW  TRIANGLE  ***** 

CALL  VORG (MARK( 6 ) ) 

CALL  VLINES(1,12,718,312,718) 

C  ****  DRAW  THE  RESPONSES  **** 

DO  900  1=1,3 

SCR( I )= VALUE (RE SPON( I , INDEX) ) 

J=-  1 

IF(SCR(  D.GE.O)  J=1 
ISCR=IABS(SCR( I ) ) 

1 1  =  1  SCR/ 4 

I J=MOD ( I  SCR, 4 )  [ 

!  IF(II.EQ.O)  GO  TO  800  ! 
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!  DO  700  K=1,II 

CALL  VCHAR(12+(I-1)*100,725+J*K*5,PTS(5),4) 
i  700  CONTINUE 

800  CALL  VCHAR( 12+(I-1)*100,725+J*(II+1)*5,PTS(IJ+1),4) 

900  CONTINUE 

CALL  VCHAR( 12,850,'  A  ',4) 

CALL  VCHAR( 1 12,850,  '  8  ',4) 

CALL  VCHAR( 212,850,  '  C  ',4) 

CALL  FORMAT (1,4, 0,SCR( 1 ) , ITERM) 

CALL  VCHAR( 12,600, ITERM, 4) 

CALL  FORMAT (1,4,0,SCR(2),ITERM) 

CALL  VCHAR( 1 12,600, ITERM, 4) 

CALL  FORMAT (1,4,0,SCR(3),ITERM) 

CALL  VCHAR( 2 1 2, 600, I  TERM, 4 ) 

CALL  VPOSIT (MARK ( 7 ) ) 

C  ****  DRAW  THE  TRIANGLE  **** 

1000  CALL  VJLINE(3,TRIX,TRIY) 

CALL  VCHAR(TRIX(1),TRIY(1)+24,'A',1) 

CALL  VCHAR( TRIX(2)-24,TRIY(2)-12, 'B  ', 1 ) 

CALL  VCHAR(TRIX(3)  +  24,TRIY(3)-12, 'C',1  ) 
UNIT=FLOAT(TRIX(3)-TRIX(2))*( SORT (3. ) /2 . ) 

D1=RESP0N( 1 , INDEX)*UNIT 
D2=RESPQN(2, INDEX)*UNIT 
D3=RESPON(3, INDEX)*UNIT 

IF((D1,EQ.0,).AND.(D2.EQ.0.). AND. (D3.EQ.0,))  GO  TO  1100 
I XO=(D 1 *. 5+D3 ) / .866+TRIX(2) 

I  YO=D 1  +  TRI Y( 2 ) 

CALL  VCHAR(IX0,IY0,'X',1) 

1 100  CALL  VPOSIT (MARK ( 8 ) ) 
c  *****  SEND  THE  TEST  PICTURE  ***** 

11200  CALL  VORG(MARK( 8  ) ) 

I  DO  1300  1=1, N 

IF((RESPON(  1,1) .EQ.O. ).AND. (RESPON( 2 , I ) . EQ. 0. ) . AND. 

X  (RESPOND, IKEQ.O.))  GO  TO  1400 
M  300  CONTINUE 

CALL  VCHAR( BOXX (1,4)  +  12,B0XY(1,4)  +  60,  'SCORE', 5) 

CALL  VCHAR(BOXX( 1 , 4 )  + 1 2 , BOXY( 1 , 4 )+36,  'PAGE', 4) 

M0DE=1 
GO  TO  1410 

1400  CALL  VCHAR( BOXX ( 1 , 4) +36, BOXY( 1 , 4 )+60,  'TEST', 4) 

CALL  VCHAR(BOXX(1,4)  +  24,BOXY(1,4)+36, 'SELECT', 6) 

MQDE=0 

1410  CALL  VPOSIT (MARK ( 9 ) ) 

I  CALL  VTRUNC(MARK(9) ) 

CALL  VSEND 

C  ****  WAIT  FOR  THE  LIGHT  PEN  **** 

:  1500  CALL  PEN(IDATA, BOXX, BOXY, NARRAY,NBOX) 

GO  TO  (1800,2100,2200,2600,2800),  NBOX 
c  ****  PROCESS  THE  TRIANGLE  **** 

C  *****  SET  UP  TRIANGULAR  UNIT  ******** 
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i  UNIT=TRIX(3)-TRIX(2)  i 

!  X=FLOAT( IDATA( 1 )-TRIX(2))/UNlT 

Y=FL0AT(IDATA(2)-TRIV(2))/UNIT 
D 1  =  ( S  QRT ( 3 . ) *  Y ) / 3 . 

D2=1.-D1-X 

D3=X-D1 

D1=D1*2. 

IF (D 1 . LT. (- .025 ) )  GO  TO  1500 
I  IF(D2.LT.(-.025))  GO  TO  1500 

IF(D3.LT.(-.025))  GO  TO  1500 
I F ( LOCKED ( INDEX) .EQ.O)  GO  TO  1600 
CALL  V0RG(MARK(9)) 

CALL  VCHAR( TRIX(2)+48,TRIY(2)+24, THEANS, 36) 

CALL  VPOS IT ( I  TERM) 

CALL  VTRUNC ( I  TERM)  ! 

CALL  VSEND 
GO  TO  1500 

1600  IFCD1.LT. (.025))  D1=0. 

IFCD2.LT. (.025))  D2=0. 

IFCD3.LT. (.025))  D3=0. 

TSUM=D1 +D2+D3 
D 1  =D  1 /TSUM 
D2=D2/TSUM 
03=03/ TSUH 

IF (  (ABSCD1-.3333) .GT..05) 

X  .OR.  (ABSCD2-.3333) .GT..05) 

X  .OR.  (ABSCD3-.3333) .GT..05))  GO  TO  1700 
D 1=1 ,/3. 

D2=D1 
D3=D  1 

1 700  RESPONC 1 t INDEX)=D1 
RESPONC  2,1 NDEX)=D2 
RESPONC  3 , 1 NDEX)=D3 
GO  TO  600 

C  ****  PROCESS  'NEXT'  BOX  **** 

1800  I NDEX=I NDEX+ 1 

IF(INDEX.LE.N)  GO  TO  100 
DO  1900  1=1, N 

IF ((RESPONC 1 , I ).EQ.O.). AND. (RESPONC 2,1). EQ.O.). AND. 

X  (RESPOND, I). EQ.O.))  GO  TO  2000 
1900  CONTINUE 
2000  INDEX=I 
i  GO  TO  1 00 

C  ****  PROCESS  'PREVIOUS'  BOX  **** 

2100  IF( INDEX. EQ.1)  GO  TO  1500 
I NDEX=I NDEX- 1 
GO  TO  100 

C  *****  PROCESS  THE  'ANSWER'  BOX  ***** 

|2200  CONTINUE  I 

IF((RESPON( 1, INDEX). EQ.O.). AND. (RESP0N(2, INDEX) .EQ.Oi ). AND. 
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X  (RESPOND, INDEX). EQ.O. ))  GO  TO  1500 
LOCKED ( INDEX)=1 
CALL  VORG(MARK{ 7 ) ) 

IX(1)=(ANSW( INDEX)- 1)*1 00 
IX(2)=IX( 1 )+70 
IX(3)=IX(2) 

I X<  4 )  =  I X( 1 ) 

IX( 5 )=IX( 1 ) 

I Y ( 1 )=850-20 
I Y( 2)=I Y( 1 ) 

IY(3)=IY(1)-250 

IY(4)=IY(3) 

IY(5)=IY(1) 

CALL  VJLI NE (4, IX, IY) 

POTU=FLOAT(MAXPLT)*ABS( VALUE ( 1 .  ) ) 

POTL=FLOAT(MAXPLT)*ABS( VALUE (0.  ) ) 

TOTAL=POTU+PQTL 

I0RGX=0 

I0RGY=0 

I XUNI T=725 

AXUNI T=IXUNI  T 

I YUNI T=525 

AYUNI T=I YUNI T 

IX( 1 )=I ORGX+ I XUNI T 

IX(2)=I0RGX 

IX(3)=I0RGX 

IX(4)=IX( 1 ) 

IX(5)=IX(1) 

IX(6)-IQRGX 
IX(7)=I0RGX 
IX(8)=IX( 1 ) 

I Y ( 1 )=IORGY 

I Y( 2)  =  I ORGY+ IF  I X ( POTL*AYUNI T/ TOTAL) 

I Y ( 3 )=I ORGY 
IY(4)=I0RGY 
IY(5)=I0RGY+I YUNIT 
IY(6)=1Y(S) 

IY(7)=IY(2) 

I Y( 8)=I Y(5) 

CALL  VLTYPE ( 2) 

CALL  VJLI NE( 7, 1 X, I Y) 

CALL  VLTYPE ( 1 ) 

DY1 =FLOAT ((IY(5)-IY(2))*1 4)/AXUNI T 
DY2=FL0AT ((IY(2)-IY(1 ))*14)/AXUNIT 
DO  2300  J=  1 ,  T4 
AJ=FLOAT(J) 

IDY1=AJ*DY1 

IDY2=AJ*DY2 

CALL  VCHAR(I0RGX+14*J,IY(2)+14+IDY1,MMAX(J),1) 
CALL  VCHAR( I ORGX+ 14*J,IY(2)-14-IDY2,MMIN(J),1) 
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12300  CONTINUE 

I X ( 1 )=IORGX 

I Y( 1 )  =  I ORGY+ IF  I X (POTL*AYUNI T/ TOTAL) 

NPL0T=0 
DO  2400  J=1  ,N 

IF ( (RESPON( 1 ,J) .EQ.O.J.AND. (RESPON( 2, J ) .EQ. 0. ) . AND. 
X  (RESP0N(3,J).EQ.0. ))  GO  TO  2400 
NPLOT=NPLOT+I 

ZZ=VALUE(RESPON(ANSW( J) ,  J) ) 

POTU=POTU-ZZ 

POTL=POTL+ZZ 

I X( NPLOT+ 1 )=IORGX+(NPLOT* IXUNI T) /MAXPLT 
I Y( NPLOT+ 1 )=IORGY+IF I X(P0TL*AYUNI T/ TOTAL ) 

2400  CONTINUE 

CALL  VJLI NE ( NPLOT, IX, I Y) 

IX( 1 )=I X( NPLOT+ 1 )+ 1 2 
IY(1)=IY( NPLOT+ 1 ) 

IX(2)=IX( l)+24 
IY(2)=IY(1) 

CALL  VCHAR(IX(I ),IY(1), '+',!) 

CALL  VJLINE(1,IX,IY) 

CALL  VCHAR(IX(2)+24, IY( 1 )  + 1  2 ,  'YOUR ',4) 

CALL  VCHAR( I X ( 2 )  +  24 , I Y ( 1 )- 1 2,  'POINTS ',6) 

IF  (NPLOT. NE.N)  GO  TO  2500 
IX( 1 )=IX( NPLOT+ 1 ) 

IX(2)=IX( 1 ) 

I Y ( 1 )  =  I  ORGY 
IY(2)=IY( 1 J+IYUNIT 
CALL  VJLINE(1,IX,IY) 

CALL  VCHAR(IX(I )-8, I Y(2)+12, 'END  ',3) 

2500  CALL  VPOSI T (MARK ( 8 ) ) 

GO  TO  1200 

c  *****  PROCESS  THE  'SELECT'  BOX  ***** 

.2600  IF(MODE.EQ.O)  GO  TO  2700 

0  **************  qq  jo  SCORES  ******************** 

N£XT=3 
j  RETURN 

q  **************  qo  TO  TEST  SELECT  *************** 

*2700  NEXT= 1 
i  RETURN 

C  *****  PRODUCE  HARDCOPY  ************** 

12800  CALL  VCOPY( 2) 
i  GO  TO  1500 

END 
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SCORING  SUBROUTINE 


C 


C 


C 


SUBROUT I NE  PRI NT ( /MO IM/ ,/INDEX/, /N/ , /RESPON/ , / ANS W/ , /MARK/ , 

X  /BLANKS/, /TEST/, /SUBJCT/,/GEOM/ ,/PVAR/, /SCORE/, 

X  /SREAL/,/TREAL/,/SBASE/,/TBASE/ ,/NEXT/ ) 

DIMENSION  RESPON ( 3 ,MDIM) ,MARK ( 1 ) , TEST ( 1 ),SUBJCT ( 1 ) , BLANKS( 1 ) 
DIMENSION  ANSW(MDIM) 

************  DIMENSION  STATEMENTS  ****************** 

DIMENSION  PARI ( 1  I ),PAR2( 33) ,PAR3 ( 13) 

DIMENSION  PAR4( 1 6) ,PAR5 (3 1 ) 

DIMENSION  IDATA( 2) 

DIMENSION  SC (3) ,LOC (4) 

DIMENSION  BOXX(5,5),BOXY(5,5) 

DIMENSION  XFRO(7),XTO(7),YFRO(7),YTO(7) 

DIMENSION  MUCHM(3) ,MUCHL(3) 

DIMENSION  SLEEP (11) ,FOOL (11) 

DIMENSION  CONS( 3 ) ,SOME (2),OVER(3), UNDER (3 ) ,KNOW( 4 ) 

DIMENSION  RE SI ( 1 0) , RES2 ( 4 ) , RE  S3 ( 2) ,RES4 ( 1 2 ) 

DIMENSION  TEMPR( 3 ) , CHAR ( 3 ) , ASTER (3 ) 

************  TYPE  STATEMENTS  *********************** 


INTEGER  SLEEP, FOOL, ASTER 

INTEGER  ANSW, MARK, BLANKS, TEST, SUBJCT, SCORE, ISCORE 

INTEGER  PAR 2, PAR 3 

INTEGER  PAR4,PAR5 

INTEGER  RES1 ,R£S2,RES3,RES4 

INTEGER  XFRO, XTO, YFRO, YTO 

INTEGER  MUCHM, MUCH, MUCHL, LESS 

INTEGER  EASY, HARD 

I NTEGER  CONS, SOME , OVER, UNDER, KNOW 

INTEGER  IDATAY, NARRAY 

INTEGER  COORXF,COORYT 

INTEGER  SC, CHAR, XSC ORE 

INTEGER  BOXX, BOXY 

REAL  HMAX 

tick *1* is &&&*&&&&  DATA  STATEMENTS  i:***********'*********^^ 
DATA  XORG/300./ , Y0RG/500. / , XUNIT/4 00. /,YUN IT/400./ 
DATA  COORXF/ 50/ , COORYT/ 450/ 

DATA  SC/  'SCOR',  'E-  ','  7 

DATA  CHAR/ 'A  ','B  '  ,'C  7 

DATA  SLEEP/  'U  ','N  ','D  ','E  ','R 

X  'V  ','A  ','L  7'U  \'E  7 

DATA  FOOL  /'O  ','V  ','E  ','R 

X  'V  ','A  ','L  ','U  ','E  ' 

DATA  BOXX/875, 975^975, 875, 875, 

X  875,975,975,875,875, 

X  875,975,975,875,875, 

X  875,975,975,875,875, 
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DATA 

X 

X 

X 

X 

DATA 

DATA 

DATA 

DATA 

DATA 

DATA 

DATA 

DATA 

DATA 

DATA 

DATA 

X 

DATA 

DATA 

DATA 

X 

DATA 

X 

DATA 

X 

X 

X 

X 

X 

DATA 

DATA 

DATA 

X 

X 

X 

X 

X 

DATA 

DATA 


0,10, 10,0,0/ 

BOXY/ 10,10,85,85, 10, 

110,116,185,185,110, 

210,210,285,285,210, 

310,310,385,385,310, 

990,990,999,999,990/ 

EASY/ 'EASY'/, HARD/ 'HARD '/ 

MUCHM  /  '  MUC',  'H  HO',  'RE  7 
MORE  /'MORE'/ 

MUCHL  /'  MUC', 'H  LE','SS  7 
LESS  /'LESS'/ 

SOME/ 'SOME', 'WHAT'/ 

CONS/ 'CONS ', 'IDER', 'ABLY'/ 

OVER/'  OVE','R  VA','LUE  7 
UNDER/'  UND','ER  V','ALUE'/ 

KNOW/'  YOU', 'R  KN', 'OWLE', 'DGE. 7 
RESI/'BY  M','ORE  ',  'REAL',  ' I STI  ', 
'  YOU',  'R  KN', 'OWLE', 'DGE. '/ 
RES2/'BY  M','ORE  ','STUD','Y.  7 

RES3/ 'OVER ' , 'ALL.'/ 

RES4/'  HEA', 'VILY',  '  ON  ',  'ANSW', 
'FEEL','  ARE','  COR','RECT', 
PARI/ 'YOU  ','CAN  ', 'IMPR','0VE  ', 
'RE  ' ,  '8 Y  ','  ','  POI', 

PAR2  /'THIS','  TES  ',  'T  W',  'AS  ' 
'  ',  '  FOR',  '  YOU',  '  THA' 

'IMPO',  'SS IB ' ,  'LE  T',  '0  DE' 
'INE  ','HOW  ',  'DISC',  'RIMI  ' 
'YOU  'ARE  ', 'AT  V',  'ALUI ' 
'OUR  ', 'FACT'.'S  AN', 'D  RE' 
PAR3  /'YOU  ',  'TEND  ',  '  TO  ',10*' 
PAR4  /'BET  ',15*'  7 

PAR5  /'THE  ',  'WAY  ','YOU  ','PLAC' 
'OUR  ', 'BETS','  ON  ','THIS 

'show',  's  yo','u  ar'.'e  ve 
'ccur',  'ate  ','in  a','sses 

'THE  ', 'VALI ', 'DITY', '  OF 
'  FAC',  'TS  A',  'ND  R',  'EASO 
ISKS/'**  7 
NARRAY/5/ 


'C  US','E  OF 


'ERS  ','YOU 


'YOUR',  ' 
'NTS  '/ 

, 'SO  ', 

,7  it;, 

, 'TERM  , 

, 'NAT I  ', 
,'NG  Y', 
, 'ASON  , 

7 


SCO 


'  IS 
'NG 

'S. 


, 'ED  Y' 
', '  TES 
','RY  A 
', 'SING 
', 'YOUR 
', 'NS. 


♦ 

J 


'T 


9 


7 


9 


Q  *******  J********************************** 

5000  FORMAT ( '1 ',' TE ST2', 5A4,5X, 'SUBJECT2  ',16A4) 

5100  FORMAT (  'O', 'DATE  2 ', 2A4, 5X,  'TIME2  ',2A4) 

5200  FORMAT ( 'O', 'SCORE  2 ', 15, 5X,  'REVISED  SCORE2 ',15, 

X  5X, 'STEREOTYPE  SCORE2 ', 15) 

5300  FORMAT ( 'O', ' I NFORMAT I ON 2 ',F6.2,8X, 'REVISED  INFORMATION2  ',F6.2 
5400  FORMAT! 'O', 'REALISM  SLOPE2 ',F7. 2, 5X, 'REALISM  INTERCEPT2', 

X  F7.2) 

5500  FORMAT!  '0',6X,  'PROBABIL I  TIES ',  23X,  'SCORES') 

5600  FORMAT!'  ',  3X,  'A  ',  7X,  'B  ',  7X,  'C  ',  1  3X,  'A  ',  1 0X,  'B  ',  1  OX,  C  ) 
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5700  FORMAT ( *  ',F6.2,A2,F6.2,A2,F6.2,A2, 

X  3X,F9.2,A2,F9.2,A2,F9.2,A2) 

5800  FORMAT ('O' , 'BASE  SLOPE : \F7. 2, 8X,  BASE  INTERCEPT:  , 

X  f  7  •  2 ) 

5900  FORMAT* '0  VBIAS:',F5. 2, 16X,  "PROB-  VARIANCE: \F6.  2) 
c  **********  THE  scoring  function  ************************ 

VALUE ( X ) =5  0 . *AL  0G 1 0 ( 3 . *  AM AX 1 (.01 ,X)) 
c  ********  DO  THE  REALISM  BIT  ******************************** 
CALL  R£AL(MDIM, N,RESPON, ANSW, SCORE, GEOM, 

X  SREAL, TREAL, SBASE , TBASE,PVAR) 

c  ********  DISPLAY  TO  INFORMATION  BOX  ********************** 
XFRO( 1 )=XORG 
XTO( 1 )=XORG+XUNI  T 
YFR0(1)=Y0RG 
YTO( 1 )=Y0RG 
XFRO( 2 ) =XTO(  1 ) 

XTQ( 2)=XT0{ 1 ) 

YFRO( 2 ) =YORG 
YTO( 2 ) =YORG+YUNI T 
XFR0(3)=XT0(1) 

XT0(3)=X0RG 

YFRO(3)=YTO(2) 

YT0(3)=YT0( 2) 

XFRO( 4) =XORG 
XT0(4)=X0RG 
YFRO( 4)=YT0(  2 ) 

YT0(4)=Y0RG 
XFR0(5 )=XORG 
XTO( 5 ) =XTO( 1 ) 

YFR0(5)=Y0RG 

YTO(5)=YTO(2) 

XFR0(6)=X0RG 
XT0(6)=XT0( 1 ) 

YFR0(6)=YT0(2) 

YT0(6)=Y0RG 

SUMA=0. 

SUMP=0. 

HMAX=ALOG10(3. )*FLOAT(N) 

A  1=0. 

PI=0. 

IF  (PVAR.LT..01 )  GO  TO  700 
DO  600  1=1, N 
DO  100  J=1 , 3 
TEMP=RE SPON ( J ,  I ) 

IF(TEMP.GT..0001 )  SUMP=SUMP+TEMP*AL0G1 0 ( TEMP ) 

100  CONTINUE 
K=  1 

DO  300  J=1 ,3 

TEMP=SREAL*RESPON( J, I )+TREAL 
IF ( TEMP. LT. 0. )  TEMP=0. 
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Vi 


I F ( TEMP.GT.  1  .  )  TEMP=U 
I F  (  J  .  NE . ANSW ( I ) )  GO  TO  200 
T£MPR( 1 )=TEMP 
GO  TO  300 
200  K=K+ 1 

TEMPR(K)=TEMP 
300  CONTINUE 

0 I FF= 1 . - TEMPR ( 1 ) 

IF ( ( TEMPR( 2) . NE.0.).0R.( TEMPR(3) . NE. 0. ) )  GO  TO  400 

TEMPR(2)=.5 

TEMPR(3)=.5 

400  DENOM=TEMPR(2)+TEMPR(3) 

TEMPR( 2) =TEMPR( 2 )*D IFF/DENOM 
TEMPR ( 3 ) =TEMPR( 3 ) *D I FF/DENOM 
DO  500  J=1 ,3 

500  IF(TEMPR(J) .GT..0001 )  SUMA=SUMA+TEMPR( J)*AL0G1 0(TEMPR(J) ) 
600  CONTINUE 

PI=HMAX+SUMP 
A I =HMAX+  SUMA 
700  CONTINUE 

XFRO( 7 ) =XORG 
XT0(7)=X-T0(  1  ) 

YFRO  ( 7 )  =  Y ORG+ ( A I /UMAX ) *  YUN I T 
YTO( 7 ) =YORG+ (PI /HMAX) * YUNI T 
CALL  VORG(MARK( 1 ) ) 

CALL  VLTYPE (2) 

CALL  VLI NES( 6, XFRO, YFRO, XTO, YTO) 

CALL  VLTYPE ( 1 ) 

CALL  VLINES( 1 , XFRO( 7 ) , YFRO ( 7 ) , XTO ( 7 ) ,  YTO ( 7 ) ) 

CALL  VCHAR(XFR0(7)-160, YFRO(7)+24, 'ACTUAL ',6) 

CALL  VCHAR(XFR0(7)-160,YFR0(7),  'KNOWLEDGE-  ', 1 0 ) 

CALL  VCHAR(XTO(7)  +  12,YTO(7)  +  24, 'PERCE I VED 9 ) 

CALL  VCHAR(XTO(7),YTO(7), '-KNOWLEDGE', 10) 

CALL  VCHAR(XFRO( 1 )+24, YFR0(3)+1 2, 

X  'COMPLETE  MASTERY',  16) 

CALL  VCHAR(XFR0(1 )+24,YFRO(1)-12, 

X  'NO  MASTERY', 10) 

I T 1 X=XFRO( 1 )+24 
IT1 Y=YFR0(3)-48 
IT2X=XFR0( 1 )+24 
IT2Y=YFR0( 1 )+48 
DO  800  1=1,11 

CALL  VCHAR( IT1X+(I-1)*12,IT1Y-(I-1)*1 2, SLEEP ( I ) , 1 ) 

CALL  VCHAR( IT1X+( 1-1 )*12, IT2Y+(I-1 ) *1 2, FOOL ( I ) , 1 ) 

800  CONTINUE 

C  **************  DISPLAY  VERBAL  RESPONSE  ************* 
BYAS=(PI-AI ) /HMAX 
CALL  DATIME ( 2,L0C) 

ITERM=Y0RG+YUNIT+75. 

CALL  VCHAR(C00RXF,ITERM,TEST,20) 
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CALL  VCHAR(CQORXF+240, I  TERM, LOC ( 1 ) , 8) 

CALL  VCHAR(C00RXF+360,  1  TERN, LOC ( 3 ) , 8) 

CALL  VCHAR(COORXF+480, 1  TERM, SUBJCT, 20) 

C  *****  ISSUE  THE  MESSAGES  ***** 

C  *****  TOO  HARD?  ***** 

IF(PVAR.GT • .  01 )  GO  TO  900 
PAR2 ( 6) =HARD 

CALL  VCHAR(COORXF,COORYT-5*24,PAR2(1 ) , 44 ) 
CALL  VCHAR( COORXF, C00RYT-6*24,PAR2( 12), 44) 
CALL  VCHAR( COORXF ,C00RYT-7*24,PAR2 (23 ) , 44) 
GO  TO  3200 

C  *****  REALLY  GOOD?  ***** 

900  IF( ABS( BYAS) •GT. ( . 1 ) )  GO  TO  1000 

CALL  VCHAR( COORXF , C00RYT-5*24,PAR5 ( 1 ) »  44 ) 
CALL  VCHAR( COORXF, COORYT- 6*24, PAR5 ( 1 2  ) , 40 ) 
CALL  VCHAR< COORXF, COORYT- 7*24, PARS (22), 40) 
GO  TO  3000 
1000  J=0 
JJ=1 

DO  1100  K=1 , 10 
1100  PAR3(3+K)=BLANKS( 1 ) 

DO  1 200  K=1 , 15 
1200  PAR4( 1+K)=BLANKS( 1 ) 

C  *****  OVER-VALUING?  ***** 

IF(BYAS.LT. (-. 1 ) )  GO  TO  2100 
IF ( BYAS. LE. ( . 7 ) )  GO  TO  1400 
DO  1300  J=1 , 3 
1300  PAR3(3+J)=C0NS( J) 

PAR4 ( 2) =MUCHL ( 1 ) 

PAR4 ( 3 ) =MUCHL ( 2 ) 

PAR4 ( 4 ) =MUCHL ( 3 ) 

JJ=3 

GO  TO  1700 

1400  IF ( BYAS.GE. ( , 4 ) )  GO  TO  1600 
DO  1500  J=l,2 
1500  PAR3 ( 3+ J ) =SOME ( J ) 

PAR4(2)=LESS 
GO  TO  1700 
1600  PAR4 ( 2 ) =LESS 
1 700  DO  1 800  K=1 ,3 
1800  PAR3 (3+J+K)=0VER(K) 

DO  1900  L=1 ,4 
1900  P AR3 ( 3+ J+K+  L ) =K NOW ( L ) 

DO  2000  K=1 , 12 
2000  PAR4( 1+JJ+K)=RES4(K) 

CALL  VCHAR(COORXF,COORYT-5*24,PAR3( 1 ),52) 
CALL  VCHAR(COORXF,COORYT-8*24,PAR4(1 ),64) 
GO  TO  3000 

C  *****  UNDER-VALUING?  ***** 

2100  IF( BYAS.GT* (-.7 ) )  GO  TO  2300 
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DO  2200  J=1 ,3 
2200  PAR3 ( 3+ J ) =CONS ( J ) 

PAR4(2)=MUCHM(1 ) 

P AR4 ( 3 ) =MUC  HM ( 2 ) 

PAR4 ( 4 ) =MUCHM ( 3 ) 

JJ=3 

GO  TO  2600 

2300  IF (BYAS.LT. (-.4) )  GO  TO  2500 
DO  2400  J=l,2 
2400  PAR3 (3+J)=S0ME( J) 

PAR4 ( 2) =MORE 
GO  TO  2600 
2500  PAR4(2)=MORE 
2600  DO  2700  K=1 ,3 
2700  PAR3 ( 3+ J+K) =UNDER(K) 

DO  2800  L=1,4 
2800  PAR3 (3+J+K+L)=KN0W(L) 

DO  2900  K=1 , 1 2 
2900  PAR4 ( 1+JJ+K)=RES4(K) 

CALL  VCHAR(COORXF,COORYT-5*24,PAR3( 1 ) , 52) 

CALL  VCHAR(COORXF,COORYT-8*24,PAR4( 1 ) ,64) 
q  *******  PRINT  THE  SCORES  ****************** 

3000  CONTINUE 
RSC0RE=0. 

DO  3100  K=1 ,N 

TEMP= ( SREAL*RESPON( ANSW( K ) , K ) +TREAL ) 

IF(TEMP.GT. 1. )  TEMP= 1 . 

IF( TEMP. LT . .  01 )  TEMP=. 0 1 
RSCORE=RSCORE+ VALUE ( TEMP ) 

3100  CONTINUE 

IF((RSCORE- SCORE). LT.O. )  RSCORE=SCORE 

I TERM=RSC ORE- SCORE* . 5 

CALL  FORMAT ( 1 ,4 , 0, I  TERM, PARI { 9) ) 

CALL  VCHAR( C OORXF,C OORYT- 1 1*24, PARI ,44) 

CALL  VCHAR(COORXF ,COORYT- 12*24, RES1 ,  40) 
ITERM=FLOAT ( N*24 )-RSC0RE+.5 
CALL  FORMAT( 1 ,4,0, ITERM,PAR1 (9) ) 

CALL  VCHAR( C OORXF,C OORYT- 14*24, PARI ,44) 

CALL  VCHAR(COORXF,C OORYT- 15*24, RE S2, 16) 

ITERM=FLOAT (N*24)-SCQRE+.5 

CALL  FORMAT ( 1 ,4 , 0, 1  TERM, PARI (9)) 

CALL  VCHAR(C00RXF,C00RYT-1  7*24, PARI ,44) 

CALL  VCHAR(COORXF,C OORYT- 18*24, RES3, 8) 

C  ****  DRAW  THE  'TEST  AND  PLOT'  BOXES  **** 

3200  CALL  VPOS I T ( MARK ( 2 ) ) 

3300  CALL  V0RG(MARK(2) ) 

CALL  VJLINE(4,B0XX( 1 , 1),B0XY( 1 , 1 )) 

CALL  VCHAR(BOXX( 1, 1 )+12,B0XY( 1 , 1 )+60, 'OLD', 3) 
CALL  VCHAR( BOXX( 1 , 1 )+1 2, BOXY (1,1 )+36, 'TEST  ',6) 
CALL  VJLINE(4,B0XX(1 ,2), BOXY (1,2)) 
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CALL  VCHAR(BOXX(1,2)+12,BOXY(1,2)+36,'PLOT  ",6) 

CALL  VJLINE(4,B0XX(1 ,3), BOXY (1,3)) 

CALL  VCHAR(BOXX( 1 ,3 )+1 2, BOXY( 1 ,3)+36,  RECORD  ,6) 
CALL  VJLINE(4,BOXX(1,4),BOXY(1,4)) 

CALL  VCHAR(BOXX( 1 , 4 )+ 12, BOXY ( 1 ,  4 )+60,  NEW  ,4) 

CALL  VCHAR(BOXX( 1 , 4)+12,BOXY( 1,4) +3 6,  'TEST", 4) 

CALL  VJLINE(4,BOXX(1,5), BOXY (1,5)) 

CALL  VPOSIT ( ITERM) 

CALL  VTRUNC( ITERM) 

CALL  VSEND 

Q  *****  *****  *****  *****  *****  *****  *****  ***** 

c  *****  WAIT  FOR  THE  LIGHT  PEN  ***************** 

3400  CALL  PEN( IDA TA, BOXX, BOXY, NARRAY, NBOX) 

GO  TO  (3500,3600,3700,4200,4300),  NBOX 
GO  TO  3400 

C  ************  RETURN  TO  OLD  TEST  ********************** 
3500  NEXT=2 
RETURN 

C  ************  GO  TO  PLOT  ROUTINE  ********************** 
3600  NEXT=4 
RETURN 

q  ************  PRINT  hardcopy  ************************** 
3  700  CALL  VORG (MARK ( 2  ) ) 

CALL  VTRUNC (MARK ( 2 ) ) 

CALL  VSEND 
TEMP=0. 

PERC1=( 1 .-SBASE-TBASE)/2. 

PERC 2=SBASE+  TBASE 
DO  3900  1=1 ,N 
PERC=PERC 1 
E=1  . 

TEM=AMAX1 (RESPON( 1 , 1 ) ,RESP0N(2, 1 ) ,RESP0N(3, 1  ) ) 
IF(ABS(RESPON(ANSW( I ) , I ) -TEM) .GT..0 1 )  GO  TO  3900 
PERC=PERC2 
E=0, 

DO  3800  J=1 ,3 

3800  IF (ABS(RESPON( J, I )-TEM) . LE..01 )  E=E+1. 

3900  TEMP=TEMP+VALUE ( PERC ) /E 
XSCORE=TEMP 

PRINT  5000, (TEST(J) , J=1 ,5), (SUBJCT( J), J=1 , 16) 

PRINT  5100, (LOC( J), J=1 , 2 ) , ( LOC( J) , J=3, 4) 
ISCORE=RSCORE 

PRINT  5200, SCORE, ISCORE , XSCORE 

TEMP1=PI/HMAX 

TEMP2=AI/HMAX 

PRINT  5300, TEMP  1 ,TEMP2 

PRINT  5400 , SREAL , TREAL 

PRINT  5800, SBASE, TBASE 

PRINT  5900,BYAS,PVAR 

PRINT  5500 
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PRINT  5600 
DO  4100  1=1, N 
DO  4000  J=1 ,3 

4000  TENPR(J)=VALUE(RESPON(J, I)) 

ASTER( 1 )=BLANKS( 1 ) 

ASTER ( 2 ) =BLANKS ( 2 ) 

ASTER ( 3 ) =BLANKS  ( 3 ) 

ASTER (ANSW( I ) )=ISKS 

PRINT  5700,RESP0N(1 , I), ASTER ( 1 ) , RESPON( 2, I ) , ASTER ( 2 ) , 
X  RESPON( 3 , I ) , ASTER ( 3 ) , TEMPR ( 1 ) , ASTER ( 1 ) , TEMPR ( 2 ) , 

X  ASTER(2),TEMPR(3),ASTER(3) 

4100  CONTINUE 

GO  TO  3300 

q  *********  get  a  new  test  ***************************** 

4200  NEXT=1 
RETURN 

q  *********  PLOT  HARDCOPY  ***************************** 

4300  CALL  VCOPY( 2 ) 

GO  TO  3400 
END 
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EXTERNAL  VALIDITY  PLOT  SUBROUTINE 


C 


C 


C 


c 

c 


SUBROUTINE  PLOT ( /MD IM/, /MARK/ , /SCORE/ , /GEOM/ , /TEST/ , 

X  /SUB JCT/ , /SREAL/ , / TREAL/ , /SBASE/ ,/TBASE/ , 

X  /N/,/RESPON/ ,/ANSW/ #/NEXT/) 

DIMENSION  MARK( 1 ) , TEST( 1 ) ,SUBJCT( 1 ) , RESPOND  ,MDIM) ,ANSW( I ) 
*********  DIMENSION  STATEMENTS  ***************************** 
DIMENSION  COORXF (5) , COORXT ( 5 ) , COORYT ( 5 ) , COOR YF (5 )  i 
DIMENSION  SC(4),XSC(4),TICS( 11 ) 

DIMENSION  HISTXF(10),HISTXT(10),HISTYF(10),HISTYT  (10) 
DIMENSION  PLOTXF ( 1 ) , PLOTXT ( 1 ),PL0TYT(1 ) ,PLOTYF ( 1 ) 

DIMENSION  BOXX( 5 » 3) »  BOXY( 5,3) 

DIMENSION  IDATA(2),L0C(4) 

*********  TYPE  DECLARATIONS  ******************************* 
INTEGER  MARK, TEST, SUBJCT, SCORE, XSCORE.ANSW 
INTEGER  COORXT, COORXF, COORYT, COORYF 
INTEGER  SC,XSC 

INTEGER  HISTXF,HISTXT,HISTYF ,HISTYT 
INTEGER  PLOTXT, PLOTXF, PLOTYT,PLOTYF 
INTEGER  BOXX, BOXY 
INTEGER  IDATA, NARRAY 

************  DATA  STATEMENTS  ****************************** 


DATA  COORXF/ 200, 200,200,200,200/ 

DATA  COORXT/ 800,200,800,200,800/ 

DATA  C0ORYF/200, 200, 200,200,200/ 

DATA  COORYT/ 2 00, 800, 200,800,800/ 

DATA  SC/'SCOR',  'E=  V  '» '  7 

DATA  XSC/'XSCO',  'RE=  ','  ',  '  7  .  .  . 

DATA  TICS/'  0  .1  .2  '  .3  .4  ,  .5 

X  ,'  .6  .7','  .8  .9  ','1.0  7 

DATA  BOXX/875,975,975,875,875, 

X  875,975,975,875,875, 

X  0,10,10,0,0/ 

DATA  BOXY/10,  10,85,85, 10, 

X  110,110,185,185,110, 

X  990,990,999,999,990/ 

DATA  NARRAY/3/ 

********  the  scoring  function  ************************* 

VALUE ( X)=50.*ALQG 1 0 ( 3.*AMAX1 (.01 ,X)) 

********  COMPUTE  THE  STEREOTYPE  SCORE  ************ 


TEMP=0. 

PERC 1=( 1 . -SBASE- TBASE )/2. 
PERC2=SBASE+TBASE 
DO  200  1=1, N 
PERC=PERC 1 


E=U 


TEM=AMAX1 (RESPON( 1 , I ) ,RESPON( 2, I ) ,RESP0N(3, I 


)) 
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IF(ABS(RESPON(ANSW(I),I)-TEM).GT,.OV)  GO  TO  200 

PERC=PERC2 

E=0. 

DO  100  J=1 , 3 

100  IF(ABS(RESP0N(J,I)-TEM).LE..01)  E=E+1. 

200  TEMP=TEMP+VALUE(PERC)/E 
XSCORE=TEMP 

C  *****  PLOT  THE  COORDINATE  SYSTEM  **** 

CALL  VORG(MARK( 1 )  ) 

CALL  F0RMAT(1, 8,0, SCORE, SC<3)) 

CALL  F ORMAT (1,8,0, XSCORE , XSC ( 3 ) ) 

CALL  VCHAR(COORXF ( 1 ) , COQRYT (2)+50,SC,l6) 

CALL  VCHAR(C00RXF(1 )+300,COORYT(2)+50,XSC,16) 

CALL  DATIME(2,L0C) 

CALL  VCHAR(C QORXF ( 1 ) , COORYT ( 2)+ 1 00, TEST ,20) 

CALL  VCHAR(COORXF ( 1 )+ 240, COORYT (2 )+ 1 00, LOC( 1 ),8) 

CALL  VCHAR(COORXF ( 1 ) +3 60, COORYT ( 2 )+ 1 00,LOC (3 ) , 8 ) 

CALL  VCHAR(C QORXF ( 1 )+480, COORYT ( 2)+ 1 00, SUBJCT, 20 ) 

CALL  VL I NES ( 5, C OORXF , COORYF , COORXT, COORYT ) 

ISTEPX=(COORXT( 1 )-COORXF( 1 ) ) / 1 0 

I  STEPY= ( COORYT( 2 ) -COORYF ( 2 ) ) / 1 0 

DO  300  J=1,11 

MIX=COORXF( 1 )+( J-1)*ISTEPX 

MIY=C00RYF(2)+( J-1 )*ISTEPY 

CALL  VCHAR(MIX,C00RYF(1  ), '1  ',  1  ) 

CALL  VCHAR(MIX- 1 2,C00RYF( 1)-36,TICS(J),4) 

CALL  VCHAR(C00RXF(2),MIY,'-',1) 

CALL  VCHAR(C00RXF(2)-60,MIY,TICS( J) ,b) 

300  CONTINUE 

C  *****  PLOT  THE  PROBABILITY  DISTRIBUTION  ***** 

DO  400  J=1 ,10 
HISTYF(J)=0 
HISTYT(J)=0 
400  CONTINUE 

DO  500  1=1, N 
DO  500  J=1 , 3 
K=RESPON(J, I)*9.99+1 . 

HISTYT(K)=HISTYT(K}+1 
IF(ANSW(I).EQ.J)  HI STYF (K)=HI$TYF (K)+l 
500  CONTINUE 

I STEPY=( COORYT ( 2 ) -COORYF ( 2 ) ) 

DO  700  J=1,10 

HISTXF ( J)=C OORXF ( 1 )+ I STEPX/2+ ( J- 1 )* ISTEPX 
HISTXT ( J)=HISTXF ( J) 

IF(HISTYT(J).EQ.O)  GO  TO  600 
I TEMP=CQQRYF( 2)+<HISTYF (J)*ISTEPY)/HISTYT(J) 

CALL  VCHAR(HISTXF( J) , I  TEMP, 'X', 1 ) 

600  HISTYT(J)=C00RYF(2)+(HISTYT(J)*ISTEPY)/(3*N) 

HI STYF ( J)=COORYF (2) 

700  CONTINUE 
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CALL  VLIN£S(10,HISTXF,HISTYF,HISTXT,HISTYT) 

C  *****  plot  THE  REALISM  LINES  **** 

UNI T=FLOAT(COORXT ( 1 ) -COORXF ( 1 ) ) 
YORG=FLOAT(COORYF( 1 ) ) 

XORG=F  LOAT ( C  OORXF ( 1  ) ) 

IF ( (TREAL. GE.O. ) .AND. (TREAL. LE. 1 . ) )  TY1=TREAL 
IF ( (SREAL.LT.O. ) .AND. (TREAL.GT. 1 . ) )  TY1=1. 
IF((SREAL.GT.O.). AND. (TREAL. LT.O.))  TY1=0. 
TEM=TREAL+SREAL 

IF ( ( TEM.GE.O. ) .AND. ( TEM.LE. 1 . ) )  TY2=TEM 
IF ((SREAL.LT.O.). AND. (TEM.LT.O. ) )  TY2=0. 

I F ( (SREAL.GT.O. ) . AND. (TEM.GT. 1 . ))  TY2=1 . 

TX 1 = ( TY 1 - TREAL ) /SRE AL 
TX2=(TY2-TREAL) /SREAL 
IF(ABS(SREAL).LT. (.01 ) )  TX1=0. 

I F ( ABS ( SREAL ).LT.(.01))  TX2=1. 

PLOTXF ( 1 )=TX 1 *UNI T+XORG 
PLOTXT ( 1 )=TX2*UN I T+XORG 
PLOTYF( 1 )=TY1*UNIT+Y0RG 
PLOTYT ( 1 )=TY2*UNIT+Y0RG 

CALL  VLINES( 1, PLOTXF, PLOTYF, PLOTXT, PLOTYT) 

CALL  VLINES( 1 , PLOTXF, PLOTYF, PLOTXT, PLOTYT) 

I F ( ( TBASE.GE.O. ) .AND. (TBASE.LE. 1 . ) )  TY1=TBASE 
IF ( (SBASE.LT.O. ) .AND. (TBASE.GT. 1 . ))  TY1=1. 

IF ( (SBASE.GT.O. ) .AND. (TBASE. LT. 0. ) )  TY1=0. 
TEM=TBASE+SBASE 

IF (( TEM. GE. 0. ) .AND. (TEM.LE. 1 . ))  TY2=TEM 
IF ((SBASE.LT.O.). AND. (TEM.LT.O.))  TY2=0. 

IF ( (SBASE.GT.O. ) .AND. ( TEM.GT. 1 . ) )  TY2=1 . 

TX1 =( TY1 -TBASE )/SBASE 
TX2=(TY2- TBASE) /SBASE 
IF( ABS( SBASE) .LT.(.OI))  TX1=0. 

IF (ABS ( SBASE ).LT.(.01))  TX2=1. 

PLOTXF ( 1 )=TX1*UN I T+XORG 
PLOTXT ( 1 )=TX2*UNI T+XORG 
PLOTYF ( 1 )=TY1*UNIT+Y0RG 
PLOTYT ( 1 )=TY2*UNI T+YORG 
CALL  VLTYPE ( 2) 

CALL  VLI NES( 1 , PLOTXF, PLOTYF, PLOTXT, PLOTYT) 

CALL  VLTYPE ( 1 ) 

C  ****  DRAW  THE  'TEST  AND  PLOT'  BOXES  **** 

CALL  VJLINE(4,B0XX( 1 , 1 ) ,BOXY( 1 , 1  )) 

CALL  VCHAR(BOXX( 1 , 1 )+12, BOXY ( 1 , 1 )+36, 'TEST  ',6) 
CALL  VJLI NE ( 4,B0XX( 1 , 2) , BOXY ( 1 , 2 ) ) 

CALL  VCHAR( BOXX( 1 , 2)  + 1 2, BOXY( 1 , 2)+60, 'SCORE', 5) 
CALL  VCHAR(BOXX( 1 , 2)+1 2, BOXY ( 1 , 2)+36, 'PAGE', 4) 
CALL  VJLI NE ( 4,B0XX( 1 , 3) , BOXY ( 1 , 3 ) ) 

CALL  VPOSI T ( I  TERM) 

CALL  VTRUNC ( ITERM) 

CALL  VSEND 
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Q  *****  *****  *****  *****  *****  *****  *****  ***** 

c  *****  WAIT  FOR  THE  LIGHT  PEN  ***************** 
800  CALL  PEN( IDATA, BOXX, BOXY, NARRAY, NBOX) 

GO  TO  (900,1000,1100),  NBOX 
GO  TO  800 
900  NEXT=2 
RETURN 
1000  NEXT=3 
RETURN 

1100  CALL  VCOPV ( 2 ) 

GO  TO  800 
END 


-67- 


REALISM  PARAMETER  SUBROUTINE 


SUBROUTINE  REAL ( /MD IM/, /N/,/RESPON/ ,/ANSW/, /SCORE/, 

X  /  GE  OM/ ,/  SRE  AL/  ,/  IRE  AL/ ,/  S  BASE/ ,/  T  BAS  E  /  , 

X  /PVAR/) 

DIMENSION  RESPON(3,MDIM) ,ANSW( 1 ) 
q  **************  TYPE  DECLARATIONS  *************************** 
INTEGER  ANSW, SCORE 

q  ********  j(-|£  SCORING  FUNCTION  ************************** 
VALUE ( X)=50.*AL0G 1 0( 3. *AMAX1 (.01 ,X)) 

C  ********  COMPUTE  THE  REALISM  VALUES  ******************* 
TSUME=0. 

TSUMK=0. 

TSUMPR=0. 

TSUMP2=0. 

DO  400  1  =  1  ,  N 
E=0. 

DO  100  J=1 ,3 

TSUMP2=TSUMP2+RESP0N( J,  I  )**2 
100  CONTINUE 

TEM=AMAX1 ( RE SPUN (1 , I ) ,RESPON( 2, I ),RESP0N(3, I ) ) 

IF ( ABS (RESPON(ANSW( I ) , I ) -TEM) .GT. (.01))  GOTO  300 
DO  200  J= 1 , 3 

IF(ABS(RESPON(J, I )- TEM) . LE . ( . 0  1  ) )  E=E+1 . 

200  CONTINUE 

300  IF(E.NE.O.)  TSUME=TSUME+1./E 
T  SUMK=  T  S  UMK+3 . 

T  SUMPR=  T  SUMPR+RE  SPON ( ANS  W( I ) , I ) 

400  CONTINUE 
TSC0RE=0. 

GEOM= 1 . 

DO  500  J=1 , N 

TSCORE=TSCORE+ VALUE ( RESPON( ANSW( J ) , J ) ) 
TEMP=RESPGN(ANSW(J), J) 

IF(TEMP.LT. (.01 ))  TEMP=. 0 1 
G  E  0M=  G  E  OM*  T  E  MP 
500  CONTINUE 

TN=FLOAT (N) 

GEOM=£XP( ALOG(GEOM) /TN) 

PBAR=TN/TSUMK 
DELTA=TSUMP2-TN*PBAR 
SREAL=( TSUMPR-PBAR*TN)/DELTA 
TREAL=PBAR*( 1.-SREAL) 

SBASE=(TSUME/TN-PBAR)/( 1.-PBAR) 

T8ASE=PBAR* ( 1 ,-SBASE ) 

SCORE=TSCORE 

PVAR=DELTA/TSUMK 

RETURN 

END 
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LIOHT  PEN  CONTROL  SUBROUTINE 


SUBROUTINE  PEN(/IDATA/,/BOXX/,/BOXY/,/NARRAY/,/NBOX/ ) 
DIMENSION  IDATA( 1 ),BOXX(5,NARRAY),BOXY(5,NARRAY) 
INTEGER  BOXX.BOXY 

c  ***********  WAIT  FOR  THE  LIGHT  BUTTON  ************* 

CALL  VLPEN(2,1,1 OOO, I DATA, I  TERM) 

DO  100  J=1 , NARRAY 
IF  ((IDATA(1).GE.60XX(1,J)) .AND. 

X  (IDATA(1 ).LE.B0XX(2, J)).AND. 

X  ( IDATA( 2) .GE.BOXY (2, J) ) .AND. 

X  ( IDATA( 2) .LE.B0XY(3, J) ) )  GO  TO  200 

100  CONTINUE 
J=0 

200  NBOX= J 
RETURN 
END 
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